Machine Learning Foundations·Lesson 39 of 67·advanced
Cross-validation and hyperparameter search
Get an estimate with an error bar instead of one number from one lucky split, and search a parameter grid without leaking the validation set into the model.
In this lesson · 20 min
- →Explain what cross-validation estimates and why one split is not enough
- →Use cross_val_score and cross_validate with the right splitter
- →Search hyperparameters with GridSearchCV and RandomizedSearchCV
- →Cross-validate a whole pipeline so preprocessing stays honest
- →Recognise when a nested cross-validation is needed
A single train/test split gives you one number, and that number moves depending on which rows happened to land where. Cross-validation gives you several, so you can tell a real improvement from noise.
One split is one sample
The same model on the same data ranges over several points depending on the seed. If you compare two models on one split each, you are mostly comparing the splits.
K-fold cross-validation
Split the data into k parts; train on k−1 and test on the remaining one, k times, so every row is tested exactly once:
cross_validate reports several metrics at once, and can hand back the training scores so you
can see the overfitting gap:
Training accuracy of 1.0 against test accuracy well below it is the tree memorising, visible in one table.
Cross-validating a pipeline
This is the part that matters most. Passing a Pipeline to cross_val_score re-fits the
preprocessing inside every fold, on that fold’s training rows only:
Grid search
GridSearchCV cross-validates every combination of the settings you give it, then refits the
best on all the data:
The double underscore is how you address a parameter inside a step: classify__C is the C of
the step named classify. It nests as deep as your pipeline does.
Look at std_test_score before celebrating: when the standard deviations overlap, the ranking is
noise and the top setting is not meaningfully better than the second.
Randomised search
With many parameters, a grid explodes combinatorially. Sampling a fixed budget usually finds an equally good setting for a fraction of the work:
Nested cross-validation
When you both tune and estimate performance, the tuning has to happen inside the outer loop, or the estimate is optimistic:
Try it yourself
Cross-validate a scaler-plus-logistic-regression pipeline with 5 stratified, shuffled folds seeded 0. Put the fold scores in scores and their mean in mean_score.
Grid-search C over the given values with 5 stratified shuffled folds seeded 0, scoring on f1. Put the search in search, the winning C in best_c.
Check yourself
You select the 10 best-correlated features from the whole dataset, then cross-validate. Why is the score inflated?
Check yourself
Five folds give accuracies of 0.81, 0.79, 0.86, 0.78 and 0.83. A second model averages 0.83 with a similar spread. What can you conclude?
Recap
- One split gives one noisy number; k-fold gives a mean and a spread.
StratifiedKFold(shuffle=True, random_state=...)for classification, always.cross_validatereports several metrics and can return training scores, which is where the overfitting gap shows.- Cross-validate the pipeline, so scaling, imputation and feature selection are re-fitted inside every fold.
GridSearchCVaddresses a step’s parameter asstep__parameter; checkstd_test_scorebefore believing the ranking.RandomizedSearchCVcovers a large space on a fixed budget.- Tuning and estimating with the same folds is optimistic; nest the search inside an outer loop for an honest figure.