Skip to content

Machine Learning Foundations·Lesson 39 of 67·advanced

Cross-validation and hyperparameter search

Get an estimate with an error bar instead of one number from one lucky split, and search a parameter grid without leaking the validation set into the model.


In this lesson · 20 min

  • →Explain what cross-validation estimates and why one split is not enough
  • →Use cross_val_score and cross_validate with the right splitter
  • →Search hyperparameters with GridSearchCV and RandomizedSearchCV
  • →Cross-validate a whole pipeline so preprocessing stays honest
  • →Recognise when a nested cross-validation is needed

A single train/test split gives you one number, and that number moves depending on which rows happened to land where. Cross-validation gives you several, so you can tell a real improvement from noise.

One split is one sample

PythonPython not started

The same model on the same data ranges over several points depending on the seed. If you compare two models on one split each, you are mostly comparing the splits.

K-fold cross-validation

Split the data into k parts; train on k−1 and test on the remaining one, k times, so every row is tested exactly once:

PythonPython not started

cross_validate reports several metrics at once, and can hand back the training scores so you can see the overfitting gap:

PythonPython not started

Training accuracy of 1.0 against test accuracy well below it is the tree memorising, visible in one table.

Cross-validating a pipeline

This is the part that matters most. Passing a Pipeline to cross_val_score re-fits the preprocessing inside every fold, on that fold’s training rows only:

PythonPython not started

GridSearchCV cross-validates every combination of the settings you give it, then refits the best on all the data:

PythonPython not started

The double underscore is how you address a parameter inside a step: classify__C is the C of the step named classify. It nests as deep as your pipeline does.

PythonPython not started

Look at std_test_score before celebrating: when the standard deviations overlap, the ranking is noise and the top setting is not meaningfully better than the second.

With many parameters, a grid explodes combinatorially. Sampling a fixed budget usually finds an equally good setting for a fraction of the work:

PythonPython not started

Nested cross-validation

When you both tune and estimate performance, the tuning has to happen inside the outer loop, or the estimate is optimistic:

PythonPython not started

Try it yourself

Exercise 1

Cross-validate a scaler-plus-logistic-regression pipeline with 5 stratified, shuffled folds seeded 0. Put the fold scores in scores and their mean in mean_score.

Your turnPython not started
Exercise 2

Grid-search C over the given values with 5 stratified shuffled folds seeded 0, scoring on f1. Put the search in search, the winning C in best_c.

Your turnPython not started

Check yourself

You select the 10 best-correlated features from the whole dataset, then cross-validate. Why is the score inflated?

Check yourself

Five folds give accuracies of 0.81, 0.79, 0.86, 0.78 and 0.83. A second model averages 0.83 with a similar spread. What can you conclude?

Recap

  • One split gives one noisy number; k-fold gives a mean and a spread.
  • StratifiedKFold(shuffle=True, random_state=...) for classification, always.
  • cross_validate reports several metrics and can return training scores, which is where the overfitting gap shows.
  • Cross-validate the pipeline, so scaling, imputation and feature selection are re-fitted inside every fold.
  • GridSearchCV addresses a step’s parameter as step__parameter; check std_test_score before believing the ranking.
  • RandomizedSearchCV covers a large space on a fixed budget.
  • Tuning and estimating with the same folds is optimistic; nest the search inside an outer loop for an honest figure.