Machine Learning Foundations·Lesson 43 of 67·advanced
PyTorch: tensors, autograd and a training loop
Build a working autograd engine in eighty lines so nothing about PyTorch is mysterious, then read the real training loop you will run on your own machine.
In this lesson · 24 min
- →Explain what a tensor adds over a NumPy array
- →Build a reverse-mode autograd engine and see backward() work
- →Recognise the five lines every PyTorch training loop contains
- →Read nn.Module, an optimiser and a loss function
- →Run the real thing locally, including on a GPU
What a tensor adds
A tensor is an array that remembers where it came from. That is the entire difference, and it is
what makes loss.backward() possible: every operation records enough to send a gradient back
through it.
c.backward() walked back through the single @ and filled in both gradients. Nothing was
written by hand — the operation recorded how to reverse itself when it happened.
The graph
Every operation adds a node. backward() sorts them topologically and visits them in reverse, so
each node’s gradient is complete before it passes anything on:
Note that x is used twice and its gradient accumulates from both paths. That accumulation is
why PyTorch needs zero_grad(): gradients add up by design, and forgetting to clear them silently
sums the gradients of every batch you have ever seen.
Training with it
Now the same two-moons network as the last lesson, written against the autograd engine. Compare this with the hand-derived backward pass — the forward pass is all that is left:
Five lines carry the whole loop, and they are the same five in every PyTorch program ever written:
logits = model(inputs) forward
loss = criterion(logits, y) measure
optimizer.zero_grad() clear last step's gradients
loss.backward() fill in this step's
optimizer.step() move the parameters
The real thing
Here is the identical network in PyTorch. Run it on your own machine:
import torch
import torch.nn as nn
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
X, y = make_moons(n_samples=600, noise=0.2, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=0, stratify=y
)
torch.manual_seed(0)
inputs = torch.tensor(X_train, dtype=torch.float32)
targets = torch.tensor(y_train, dtype=torch.long)
model = nn.Sequential(
nn.Linear(2, 16),
nn.ReLU(),
nn.Linear(16, 2),
)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.5)
for epoch in range(1, 401):
logits = model(inputs)
loss = criterion(logits, targets)
optimizer.zero_grad()
loss.backward()
optimizer.step()
if epoch % 100 == 0:
print(f"epoch {epoch:3} loss {loss.item():.4f}")
model.eval()
with torch.no_grad():
test_logits = model(torch.tensor(X_test, dtype=torch.float32))
accuracy = (test_logits.argmax(dim=1).numpy() == y_test).mean()
print("test accuracy:", round(float(accuracy), 4))
Four things in that listing are worth naming:
nn.Module— a container that knows its parameters.nn.Sequentialis the simplest one; writing your own means subclassingnn.Moduleand definingforward.nn.CrossEntropyLossexpects raw logits, not probabilities. It applies the log-softmax internally, for exactly the numerical-stability reasons module 3 covered. Passing it a softmax output is a real and common bug.optimizer.zero_grad()must come beforebackward(), because gradients accumulate.torch.no_grad()switches off graph recording for evaluation, which saves memory and time.
A custom module, for when Sequential is not enough:
class MoonNet(nn.Module):
def __init__(self, hidden: int = 16) -> None:
super().__init__()
self.first = nn.Linear(2, hidden)
self.second = nn.Linear(hidden, 2)
def forward(self, x: torch.Tensor) -> torch.Tensor:
return self.second(torch.relu(self.first(x)))
model = MoonNet()
print(sum(p.numel() for p in model.parameters()), "parameters")
And the parts you meet as soon as the data does not fit in one batch:
from torch.utils.data import TensorDataset, DataLoader
dataset = TensorDataset(inputs, targets)
loader = DataLoader(dataset, batch_size=32, shuffle=True)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
for epoch in range(20):
for batch_inputs, batch_targets in loader:
batch_inputs = batch_inputs.to(device)
batch_targets = batch_targets.to(device)
loss = criterion(model(batch_inputs), batch_targets)
optimizer.zero_grad()
loss.backward()
optimizer.step()
torch.save(model.state_dict(), "moonnet.pt")
Try it yourself
Write matmul_backward(grad, A, B), returning the gradients with respect to A and B for C = A @ B.
Write sgd_step(parameters, learning_rate): move each parameter against its gradient, then zero that gradient — what optimizer.step() and zero_grad() do together.
Check yourself
Why does PyTorch require optimizer.zero_grad() before loss.backward()?
Check yourself
You pass softmax probabilities to nn.CrossEntropyLoss and training goes badly. Why?
Recap
- A tensor is an array that records the operations applied to it; that record is what
backward()walks. backward()topologically sorts the graph and visits it in reverse, so each node’s gradient is complete before it passes anything back.- Gradients accumulate, which is why
zero_grad()exists and why forgetting it corrupts training silently. - Every PyTorch loop is forward, loss,
zero_grad,backward,step— unchanged at any scale. nn.CrossEntropyLosstakes raw logits and class indices; wrap evaluation intorch.no_grad().- The autograd on this page is not a simplification of the idea, only of the engineering.