Skip to content

Numerical Python·Lesson 26 of 67·intermediate

Matrix calculus intuition for gradients

What a gradient is, how the chain rule composes one, and how to check an analytic gradient against a numerical one — the intuition every training loop rests on.


In this lesson · 20 min

  • →Describe a derivative as a slope, and estimate one numerically
  • →Read a gradient as the vector of partial derivatives
  • →Apply the chain rule to compose gradients through a pipeline
  • →Derive the gradient of a mean-squared-error loss and take a descent step
  • →Verify an analytic gradient with a numerical gradient check

Training a model means repeatedly asking “if I nudge this number, does the loss go up or down, and by how much?”. The answer is a gradient. You will rarely derive one by hand in practice — a framework does it — but knowing what it is turns a training loop from magic into mechanics.

A derivative is a slope

PythonPython not started

The slope is positive where the curve rises and negative where it falls, and it is zero at the minimum. That last fact is the whole idea: step in the opposite direction of the slope and the value goes down.

PythonPython not started

Gradients of several variables

With more than one input, the derivative becomes a vector of partial derivatives — one per input, each answering “if I nudge only this one, what happens?”.

PythonPython not started

The gradient points in the direction of steepest increase, so descent subtracts it:

PythonPython not started

The chain rule

A pipeline of functions has a gradient that is the product of each stage’s gradient. If y = g(f(x)), then dy/dx = dg/df * df/dx:

PythonPython not started

The gradient of a squared-error loss

This is the one worth knowing by heart, because it is the loss behind linear regression and the final layer of countless models.

For predictions p = X @ w and targets y, with loss = mean((p - y)^2):

  • d loss / d p = 2 * (p - y) / n
  • d p / d w = X
  • so d loss / d w = X.T @ (2 * (p - y) / n)
PythonPython not started

That is a complete training loop: forward pass, loss, gradient, update. Everything module 5 adds is more layers and a smarter optimiser.

Gradient checking

When you write a gradient by hand, check it against a numerical one. The numerical version is far too slow to train with — it costs two forward passes per parameter — but it is the ground truth for a bug hunt:

PythonPython not started

And here is what a wrong gradient looks like, so you know the failure when you see it:

PythonPython not started

A constant ratio of exactly 2 is the signature of a missing factor. A gradient that is wrong by a constant still trains — just at the wrong learning rate — which is why this class of bug survives so long.

Try it yourself

Exercise 1

Write numerical_gradient(f, w, eps=1e-6) using the two-sided formula, returning one partial derivative per element of w.

Your turnPython not started
Exercise 2

Complete gradient(w) for the mean squared error of X @ w against y, then take 100 descent steps at a learning rate of 0.1 into w.

Your turnPython not started

Check yourself

Your analytic gradient is consistently exactly half the numerical one. What does that tell you?

Check yourself

Why is a numerical gradient useless for actually training a model?

Recap

  • A derivative is a slope: positive going up, negative going down, zero at a minimum.
  • A gradient is the vector of partial derivatives, pointing towards steepest increase — so descent subtracts it, scaled by a learning rate.
  • The chain rule multiplies each stage’s local gradient; backpropagation is that applied to a graph, and it explains both vanishing and exploding gradients.
  • For loss = mean((X @ w - y) ** 2), the gradient is X.T @ (2 * (X @ w - y) / n).
  • Check a hand-written gradient against a two-sided numerical estimate with eps around 1e-6. A constant ratio means a missing factor, which trains anyway and hides for months.