Skip to content

Machine Learning Foundations·Lesson 36 of 67·advanced

Train, validation and test splits

Why a model must be judged on data it has never seen, how to split three ways, and the five kinds of leakage that make a held-out score a lie.


In this lesson · 18 min

  • →Split data into training and held-out sets, reproducibly and with stratification
  • →Explain what the validation set is for and why the test set is used once
  • →Recognise preprocessing leakage, duplicate leakage and target leakage
  • →Split by group when rows share a source
  • →Split by time when the data is ordered

A model’s score on the data it learned from is not a measurement — it is a memory test. Everything in this lesson exists to make sure the number you report is the number you will get in production.

Why hold anything out

PythonPython not started

A full-depth decision tree scores 1.0 on its training data every time — it memorised it. Only the second number says anything about a new example.

Splitting

PythonPython not started

Three splits, three jobs

  • Training set — the model learns from it.
  • Validation set — you choose between models and settings with it.
  • Test set — you use it once, at the end, to report a number.
PythonPython not started

Note the second test_size=0.25: a quarter of the remaining 80% is 20% of the whole.

PythonPython not started

Leakage

Leakage is any way information from outside the training data reaches the model. Its signature is a validation score that is too good, followed by disappointing production results.

Preprocessing before the split

PythonPython not started

The difference is small here and enormous on a real dataset with outliers. The rule is absolute: fit every transformer on the training data only. The next lesson’s Pipeline makes it structurally impossible to get wrong.

Duplicates across the split

PythonPython not started

Target leakage

A feature that would not exist at prediction time can make a model look superb and be useless:

PythonPython not started

The second model is nearly perfect and completely worthless: at prediction time there is no refusal message, because the refusal has not happened yet.

Splitting by group and by time

When rows share a source — several chunks from one document, several turns from one conversation — a random split puts siblings on both sides, and the model recognises the document rather than learning the task:

PythonPython not started

For ordered data, the split has to respect time — train on the past, test on the future:

PythonPython not started

Try it yourself

Exercise 1

Split into 60% train, 20% validation and 20% test, stratified by y, with random_state=0 throughout.

Your turnPython not started
Exercise 2

The scaler below is fitted on every row, which leaks the held-out data. Split first, then fit the scaler on the training set only and apply it to both.

Your turnPython not started

Check yourself

You tune ten hyperparameter settings on the validation set and report the best validation score. Why is it optimistic?

Check yourself

Your chunk-level classifier scores 0.97 held out, then 0.62 in production. Chunks come from 200 documents. What is the likely cause?

Recap

  • Score on data the model has not seen; a training score measures memorisation.
  • train_test_split(..., stratify=y, random_state=0) keeps class balance and makes the split reproducible.
  • Train to learn, validation to choose, test once to report.
  • Fit every transformer on the training data only — preprocessing before the split leaks.
  • De-duplicate on content before splitting, or the score measures recall.
  • A feature that only exists after the outcome is target leakage: excellent offline, worthless live.
  • Split by group when rows share a source, and by time when the data is ordered.