Skip to content

Machine Learning Foundations·Lesson 42 of 67·advanced

A neural network in pure NumPy

Build a two-layer classifier — forward pass, cross-entropy loss, backpropagation and a training loop — in about forty lines, with no framework involved.


In this lesson · 24 min

  • →Write the forward pass of a two-layer network
  • →Compute a stable softmax and cross-entropy loss
  • →Derive and implement the backward pass
  • →Initialise weights so signal survives the first layer
  • →Train the network and watch the decision boundary form

A neural network is a chain of matrix multiplications with a non-linearity between them. There is no magic in it, and building one by hand is the fastest way to stop finding autograd mysterious.

The problem

Two interleaving half-moons: not separable by any straight line, so a linear model cannot do it.

PythonPython not started

The architecture

X  (n, 2)
  -> Z1 = X @ W1 + b1     (n, hidden)
  -> A1 = relu(Z1)        (n, hidden)
  -> Z2 = A1 @ W2 + b2    (n, classes)     the logits
  -> P  = softmax(Z2)     (n, classes)     the probabilities

Without the ReLU, two stacked linear layers collapse into one linear layer — the non-linearity is what buys the extra capacity.

PythonPython not started

Initialisation

Start every weight at zero and every hidden unit computes the same thing for ever — there is nothing to break the symmetry. Start them too large and the activations explode.

PythonPython not started

He initialisation — a normal distribution scaled by the square root of 2 over the number of inputs — keeps the signal roughly the same size at every depth. That factor of 2 is there because ReLU throws away half of it.

The loss

Cross-entropy measures how surprised the model is by the true label:

PythonPython not started

P[np.arange(n), y] picks each row’s probability for that row’s true class — integer-array indexing from module 3, doing real work.

The backward pass

Each line is the chain rule applied to one operation:

dZ2 = (P - onehot(y)) / n        the softmax + cross-entropy gradient, together
dW2 = A1.T @ dZ2                 (hidden, classes)
db2 = dZ2.sum(axis=0)            (classes,)
dA1 = dZ2 @ W2.T                 (n, hidden)
dZ1 = dA1 * (Z1 > 0)             ReLU passes the gradient only where it was active
dW1 = X.T @ dZ1                  (features, hidden)
db1 = dZ1.sum(axis=0)            (hidden,)

The first line is the pleasant surprise: differentiating softmax and cross-entropy separately is messy, and together they collapse to prediction minus truth, exactly as in logistic regression.

PythonPython not started

Differences around 1e-10 mean the hand-written gradients are right. This check is worth running on any network you write yourself.

Training

PythonPython not started

Around 0.97 — from a linear model’s 0.87, using forty lines and no framework.

PythonPython not started

Try it yourself

Exercise 1

Implement relu(z) and relu_backward(grad, z), which passes the incoming gradient through only where z was positive.

Your turnPython not started
Exercise 2

Implement forward(X, params): linear, ReLU, linear, softmax. Return the tuple (Z1, A1, P).

Your turnPython not started

Check yourself

Why does the network need the ReLU between the two linear layers?

Check yourself

Why does the backward pass start with (P - onehot(y)) / n rather than something more complicated?

Recap

  • A network is linear, non-linear, linear, softmax — and without the non-linearity the layers collapse into one.
  • Initialise with He scaling, sqrt(2 / fan_in), so activations neither vanish nor explode with depth.
  • Cross-entropy is -log of the probability assigned to the true class; index it with P[np.arange(n), y].
  • The softmax-plus-cross-entropy gradient is (P - onehot) / n; from there the backward pass is one transposed matrix multiplication per layer.
  • ReLU’s backward pass is grad * (Z > 0), which is also why units can die.
  • Check every hand-written gradient against a numerical one before trusting it.