Machine Learning Foundations·Lesson 40 of 67·advanced
Overfitting, regularisation and the bias-variance trade-off
Watch a model memorise noise as you give it more capacity, read a learning curve to tell whether more data would help, and use regularisation to buy generalisation with a little bias.
In this lesson · 20 min
- →Recognise overfitting and underfitting from training and validation scores
- →Read a validation curve to choose model complexity
- →Read a learning curve to decide whether more data would help
- →Apply L2 and L1 regularisation, and explain what each does
- →Describe the bias-variance trade-off in terms you can act on
A model that fits its training data perfectly has usually learned the noise as well as the signal. This lesson is about seeing that happen, measuring it, and doing something about it.
Watching it happen
Fit polynomials of increasing degree to twelve noisy points:
Training error falls the whole way. Test error falls, bottoms out, then climbs — and at degree 11 the model passes exactly through all twelve points and is useless everywhere else.
- Underfitting — high error on both training and held-out data. The model cannot represent the pattern. That is bias.
- Overfitting — low training error, high held-out error. The model represents this sample rather than the pattern. That is variance.
The validation curve
Sweeping one complexity parameter and plotting both scores is how you find the turning point:
The learning curve
A learning curve answers a different question: would collecting more data help?
Read it like this:
- A wide gap that is still narrowing means high variance — more data will help.
- Both curves flat and low, close together, means high bias — more data will not help. You need a better model or better features.
Regularisation
Regularisation penalises large coefficients, which limits how wiggly a model can be:
The degree-11 model is rescued by a penalty that stops its coefficients reaching into the thousands. Two penalties are worth knowing:
- L2 (ridge) shrinks every coefficient towards zero, smoothly. Nothing quite reaches zero.
- L1 (lasso) drives many coefficients to exactly zero, which selects features.
In LogisticRegression and the support vector machines, the parameter is C, which is the
inverse of the penalty strength — small C means heavy regularisation:
The trade-off, stated plainly
Held-out error decomposes into three parts: bias (the model cannot represent the truth), variance (the model changes a lot with the sample) and irreducible noise. Adding capacity lowers bias and raises variance; regularising does the reverse.
Bias falls and variance rises with depth, and their sum has a minimum in the middle. That minimum is what every choice in this lesson is looking for.
Try it yourself
Fit a polynomial of each degree and record the held-out MSE in test_errors, then set best_degree to the degree with the lowest error.
For each alpha, fit a scaled Ridge on the training data and record (train_mse, test_mse) in results. Then set best_alpha to the one with the lowest held-out error.
Check yourself
Training accuracy is 1.00 and validation accuracy is 0.71, and the gap is not closing as you add data. What should you try?
Check yourself
What does lasso (L1) do that ridge (L2) does not?
Recap
- Underfitting is high error everywhere (bias); overfitting is low training error with high held-out error (variance).
- A validation curve over one complexity parameter shows where held-out performance turns.
- A learning curve says whether more data would help: a closing gap yes, two flat converged curves no.
- Regularisation trades a little bias for less variance. L2 shrinks smoothly; L1 zeroes coefficients and selects features. Scale the features first.
- In
LogisticRegressionand SVMs the knob isC, the inverse of the penalty. - Tuning prompts against a fixed eval set overfits in exactly the same way, and needs the same held-out discipline.