Numerical Python·Lesson 26 of 67·intermediate
Matrix calculus intuition for gradients
What a gradient is, how the chain rule composes one, and how to check an analytic gradient against a numerical one — the intuition every training loop rests on.
In this lesson · 20 min
- →Describe a derivative as a slope, and estimate one numerically
- →Read a gradient as the vector of partial derivatives
- →Apply the chain rule to compose gradients through a pipeline
- →Derive the gradient of a mean-squared-error loss and take a descent step
- →Verify an analytic gradient with a numerical gradient check
Training a model means repeatedly asking “if I nudge this number, does the loss go up or down, and by how much?”. The answer is a gradient. You will rarely derive one by hand in practice — a framework does it — but knowing what it is turns a training loop from magic into mechanics.
A derivative is a slope
The slope is positive where the curve rises and negative where it falls, and it is zero at the minimum. That last fact is the whole idea: step in the opposite direction of the slope and the value goes down.
Gradients of several variables
With more than one input, the derivative becomes a vector of partial derivatives — one per input, each answering “if I nudge only this one, what happens?”.
The gradient points in the direction of steepest increase, so descent subtracts it:
The chain rule
A pipeline of functions has a gradient that is the product of each stage’s gradient. If
y = g(f(x)), then dy/dx = dg/df * df/dx:
The gradient of a squared-error loss
This is the one worth knowing by heart, because it is the loss behind linear regression and the final layer of countless models.
For predictions p = X @ w and targets y, with loss = mean((p - y)^2):
d loss / d p = 2 * (p - y) / nd p / d w = X- so
d loss / d w = X.T @ (2 * (p - y) / n)
That is a complete training loop: forward pass, loss, gradient, update. Everything module 5 adds is more layers and a smarter optimiser.
Gradient checking
When you write a gradient by hand, check it against a numerical one. The numerical version is far too slow to train with — it costs two forward passes per parameter — but it is the ground truth for a bug hunt:
And here is what a wrong gradient looks like, so you know the failure when you see it:
A constant ratio of exactly 2 is the signature of a missing factor. A gradient that is wrong by a constant still trains — just at the wrong learning rate — which is why this class of bug survives so long.
Try it yourself
Write numerical_gradient(f, w, eps=1e-6) using the two-sided formula, returning one partial derivative per element of w.
Complete gradient(w) for the mean squared error of X @ w against y, then take 100 descent steps at a learning rate of 0.1 into w.
Check yourself
Your analytic gradient is consistently exactly half the numerical one. What does that tell you?
Check yourself
Why is a numerical gradient useless for actually training a model?
Recap
- A derivative is a slope: positive going up, negative going down, zero at a minimum.
- A gradient is the vector of partial derivatives, pointing towards steepest increase — so descent subtracts it, scaled by a learning rate.
- The chain rule multiplies each stage’s local gradient; backpropagation is that applied to a graph, and it explains both vanishing and exploding gradients.
- For
loss = mean((X @ w - y) ** 2), the gradient isX.T @ (2 * (X @ w - y) / n). - Check a hand-written gradient against a two-sided numerical estimate with
epsaround1e-6. A constant ratio means a missing factor, which trains anyway and hides for months.