Machine Learning Foundations·Lesson 42 of 67·advanced
A neural network in pure NumPy
Build a two-layer classifier — forward pass, cross-entropy loss, backpropagation and a training loop — in about forty lines, with no framework involved.
In this lesson · 24 min
- →Write the forward pass of a two-layer network
- →Compute a stable softmax and cross-entropy loss
- →Derive and implement the backward pass
- →Initialise weights so signal survives the first layer
- →Train the network and watch the decision boundary form
A neural network is a chain of matrix multiplications with a non-linearity between them. There is no magic in it, and building one by hand is the fastest way to stop finding autograd mysterious.
The problem
Two interleaving half-moons: not separable by any straight line, so a linear model cannot do it.
The architecture
X (n, 2)
-> Z1 = X @ W1 + b1 (n, hidden)
-> A1 = relu(Z1) (n, hidden)
-> Z2 = A1 @ W2 + b2 (n, classes) the logits
-> P = softmax(Z2) (n, classes) the probabilities
Without the ReLU, two stacked linear layers collapse into one linear layer — the non-linearity is what buys the extra capacity.
Initialisation
Start every weight at zero and every hidden unit computes the same thing for ever — there is nothing to break the symmetry. Start them too large and the activations explode.
He initialisation — a normal distribution scaled by the square root of 2 over the number of inputs — keeps the signal roughly the same size at every depth. That factor of 2 is there because ReLU throws away half of it.
The loss
Cross-entropy measures how surprised the model is by the true label:
P[np.arange(n), y] picks each row’s probability for that row’s true class — integer-array
indexing from module 3, doing real work.
The backward pass
Each line is the chain rule applied to one operation:
dZ2 = (P - onehot(y)) / n the softmax + cross-entropy gradient, together
dW2 = A1.T @ dZ2 (hidden, classes)
db2 = dZ2.sum(axis=0) (classes,)
dA1 = dZ2 @ W2.T (n, hidden)
dZ1 = dA1 * (Z1 > 0) ReLU passes the gradient only where it was active
dW1 = X.T @ dZ1 (features, hidden)
db1 = dZ1.sum(axis=0) (hidden,)
The first line is the pleasant surprise: differentiating softmax and cross-entropy separately is messy, and together they collapse to prediction minus truth, exactly as in logistic regression.
Differences around 1e-10 mean the hand-written gradients are right. This check is worth running
on any network you write yourself.
Training
Around 0.97 — from a linear model’s 0.87, using forty lines and no framework.
Try it yourself
Implement relu(z) and relu_backward(grad, z), which passes the incoming gradient through only where z was positive.
Implement forward(X, params): linear, ReLU, linear, softmax. Return the tuple (Z1, A1, P).
Check yourself
Why does the network need the ReLU between the two linear layers?
Check yourself
Why does the backward pass start with (P - onehot(y)) / n rather than something more complicated?
Recap
- A network is linear, non-linear, linear, softmax — and without the non-linearity the layers collapse into one.
- Initialise with He scaling,
sqrt(2 / fan_in), so activations neither vanish nor explode with depth. - Cross-entropy is
-logof the probability assigned to the true class; index it withP[np.arange(n), y]. - The softmax-plus-cross-entropy gradient is
(P - onehot) / n; from there the backward pass is one transposed matrix multiplication per layer. - ReLU’s backward pass is
grad * (Z > 0), which is also why units can die. - Check every hand-written gradient against a numerical one before trusting it.