Machine Learning Foundations·Lesson 36 of 67·advanced
Train, validation and test splits
Why a model must be judged on data it has never seen, how to split three ways, and the five kinds of leakage that make a held-out score a lie.
In this lesson · 18 min
- →Split data into training and held-out sets, reproducibly and with stratification
- →Explain what the validation set is for and why the test set is used once
- →Recognise preprocessing leakage, duplicate leakage and target leakage
- →Split by group when rows share a source
- →Split by time when the data is ordered
A model’s score on the data it learned from is not a measurement — it is a memory test. Everything in this lesson exists to make sure the number you report is the number you will get in production.
Why hold anything out
A full-depth decision tree scores 1.0 on its training data every time — it memorised it. Only the second number says anything about a new example.
Splitting
Three splits, three jobs
- Training set — the model learns from it.
- Validation set — you choose between models and settings with it.
- Test set — you use it once, at the end, to report a number.
Note the second test_size=0.25: a quarter of the remaining 80% is 20% of the whole.
Leakage
Leakage is any way information from outside the training data reaches the model. Its signature is a validation score that is too good, followed by disappointing production results.
Preprocessing before the split
The difference is small here and enormous on a real dataset with outliers. The rule is absolute:
fit every transformer on the training data only. The next lesson’s Pipeline makes it
structurally impossible to get wrong.
Duplicates across the split
Target leakage
A feature that would not exist at prediction time can make a model look superb and be useless:
The second model is nearly perfect and completely worthless: at prediction time there is no refusal message, because the refusal has not happened yet.
Splitting by group and by time
When rows share a source — several chunks from one document, several turns from one conversation — a random split puts siblings on both sides, and the model recognises the document rather than learning the task:
For ordered data, the split has to respect time — train on the past, test on the future:
Try it yourself
Split into 60% train, 20% validation and 20% test, stratified by y, with random_state=0 throughout.
The scaler below is fitted on every row, which leaks the held-out data. Split first, then fit the scaler on the training set only and apply it to both.
Check yourself
You tune ten hyperparameter settings on the validation set and report the best validation score. Why is it optimistic?
Check yourself
Your chunk-level classifier scores 0.97 held out, then 0.62 in production. Chunks come from 200 documents. What is the likely cause?
Recap
- Score on data the model has not seen; a training score measures memorisation.
train_test_split(..., stratify=y, random_state=0)keeps class balance and makes the split reproducible.- Train to learn, validation to choose, test once to report.
- Fit every transformer on the training data only — preprocessing before the split leaks.
- De-duplicate on content before splitting, or the score measures recall.
- A feature that only exists after the outcome is target leakage: excellent offline, worthless live.
- Split by group when rows share a source, and by time when the data is ordered.