Machine Learning Foundations·Lesson 41 of 67·advanced
Gradient descent from scratch
Build the optimiser every model is trained with — full-batch, stochastic and mini-batch — and see exactly what the learning rate does when you get it wrong.
In this lesson · 20 min
- →Implement full-batch gradient descent for linear regression
- →Diagnose a learning rate that is too small, too large or divergent
- →Explain why feature scaling changes how fast descent converges
- →Implement stochastic and mini-batch gradient descent
- →Add momentum, and see why it helps
Module 3 established what a gradient is. This lesson turns it into the optimiser that trains every model you will ever use, in about fifteen lines.
Full-batch gradient descent
Compute the gradient over all the data, take one step against it, repeat:
Three lines do the work: predict, measure the error, step against the gradient. Everything else in deep learning is elaboration on those three.
The learning rate
It is the one hyperparameter you cannot avoid tuning, and all three failure modes are worth seeing:
- Too small — it converges, eventually, but you will run out of patience first.
- About right — a smooth drop that flattens out.
- Too large — the loss bounces, or overshoots so badly it goes to infinity.
Scaling changes the geometry
Gradient descent follows the steepest direction, which is only the right direction when the features are on comparable scales:
With wildly different scales, a learning rate small enough to be stable for the large feature is far too small for the small one — so one weight barely moves. Standardising the features makes a single learning rate work for all of them.
Stochastic and mini-batch
Full-batch descent uses every row for each step, which is exact and, on a million rows, slow. Stochastic descent uses one row; mini-batch uses a handful, and is what everything actually does:
A small batch gives many cheap, noisy steps; a large batch gives few expensive, accurate ones. The noise is not purely a cost — it helps the optimiser escape narrow valleys.
Momentum
Plain descent zig-zags across a narrow valley. Momentum keeps a running average of recent gradients, which cancels the zig-zag and accelerates along the floor of the valley:
Logistic regression by hand
The same loop, with a different loss. For classification the loss is cross-entropy, and its gradient is remarkably tidy — the prediction minus the truth:
Try it yourself
Write descend(X, y, learning_rate, steps): start from zeros, take that many full-batch gradient-descent steps on the mean squared error, and return the weights.
Write minibatch(X, y, batch_size, epochs, learning_rate, seed): reshuffle the rows each epoch and take one gradient step per batch.
Check yourself
Your training loss drops for twenty steps and then jumps to nan. What is the most likely cause?
Check yourself
Why is mini-batch descent preferred over full-batch on a large dataset?
Recap
- Gradient descent is three lines: predict, measure the error, step against the gradient.
- Diagnose the learning rate from a log-scale loss curve — flat means too small, jagged or rising means too large.
- Standardise features before descending, or one learning rate cannot suit them all.
- Mini-batch descent reshuffles each epoch and steps per batch; small batches mean many noisy steps, large batches few accurate ones.
- Momentum averages recent gradients, cancelling the zig-zag across a narrow valley.
- The cross-entropy gradient for logistic regression is
X.T @ (prediction - truth) / n, and the logits must be clipped or the loss becomesnan.