Skip to content

Machine Learning Foundations·Lesson 41 of 67·advanced

Gradient descent from scratch

Build the optimiser every model is trained with — full-batch, stochastic and mini-batch — and see exactly what the learning rate does when you get it wrong.


In this lesson · 20 min

  • →Implement full-batch gradient descent for linear regression
  • →Diagnose a learning rate that is too small, too large or divergent
  • →Explain why feature scaling changes how fast descent converges
  • →Implement stochastic and mini-batch gradient descent
  • →Add momentum, and see why it helps

Module 3 established what a gradient is. This lesson turns it into the optimiser that trains every model you will ever use, in about fifteen lines.

Full-batch gradient descent

Compute the gradient over all the data, take one step against it, repeat:

PythonPython not started

Three lines do the work: predict, measure the error, step against the gradient. Everything else in deep learning is elaboration on those three.

The learning rate

It is the one hyperparameter you cannot avoid tuning, and all three failure modes are worth seeing:

PythonPython not started
PythonPython not started
  • Too small — it converges, eventually, but you will run out of patience first.
  • About right — a smooth drop that flattens out.
  • Too large — the loss bounces, or overshoots so badly it goes to infinity.

Scaling changes the geometry

Gradient descent follows the steepest direction, which is only the right direction when the features are on comparable scales:

PythonPython not started

With wildly different scales, a learning rate small enough to be stable for the large feature is far too small for the small one — so one weight barely moves. Standardising the features makes a single learning rate work for all of them.

Stochastic and mini-batch

Full-batch descent uses every row for each step, which is exact and, on a million rows, slow. Stochastic descent uses one row; mini-batch uses a handful, and is what everything actually does:

PythonPython not started

A small batch gives many cheap, noisy steps; a large batch gives few expensive, accurate ones. The noise is not purely a cost — it helps the optimiser escape narrow valleys.

Momentum

Plain descent zig-zags across a narrow valley. Momentum keeps a running average of recent gradients, which cancels the zig-zag and accelerates along the floor of the valley:

PythonPython not started

Logistic regression by hand

The same loop, with a different loss. For classification the loss is cross-entropy, and its gradient is remarkably tidy — the prediction minus the truth:

PythonPython not started

Try it yourself

Exercise 1

Write descend(X, y, learning_rate, steps): start from zeros, take that many full-batch gradient-descent steps on the mean squared error, and return the weights.

Your turnPython not started
Exercise 2

Write minibatch(X, y, batch_size, epochs, learning_rate, seed): reshuffle the rows each epoch and take one gradient step per batch.

Your turnPython not started

Check yourself

Your training loss drops for twenty steps and then jumps to nan. What is the most likely cause?

Check yourself

Why is mini-batch descent preferred over full-batch on a large dataset?

Recap

  • Gradient descent is three lines: predict, measure the error, step against the gradient.
  • Diagnose the learning rate from a log-scale loss curve — flat means too small, jagged or rising means too large.
  • Standardise features before descending, or one learning rate cannot suit them all.
  • Mini-batch descent reshuffles each epoch and steps per batch; small batches mean many noisy steps, large batches few accurate ones.
  • Momentum averages recent gradients, cancelling the zig-zag across a narrow valley.
  • The cross-entropy gradient for logistic regression is X.T @ (prediction - truth) / n, and the logits must be clipped or the loss becomes nan.