Skip to content

Machine Learning Foundations·Lesson 43 of 67·advanced

PyTorch: tensors, autograd and a training loop

Build a working autograd engine in eighty lines so nothing about PyTorch is mysterious, then read the real training loop you will run on your own machine.


In this lesson · 24 min

  • →Explain what a tensor adds over a NumPy array
  • →Build a reverse-mode autograd engine and see backward() work
  • →Recognise the five lines every PyTorch training loop contains
  • →Read nn.Module, an optimiser and a loss function
  • →Run the real thing locally, including on a GPU

What a tensor adds

A tensor is an array that remembers where it came from. That is the entire difference, and it is what makes loss.backward() possible: every operation records enough to send a gradient back through it.

PythonPython not started

c.backward() walked back through the single @ and filled in both gradients. Nothing was written by hand — the operation recorded how to reverse itself when it happened.

The graph

Every operation adds a node. backward() sorts them topologically and visits them in reverse, so each node’s gradient is complete before it passes anything on:

PythonPython not started

Note that x is used twice and its gradient accumulates from both paths. That accumulation is why PyTorch needs zero_grad(): gradients add up by design, and forgetting to clear them silently sums the gradients of every batch you have ever seen.

Training with it

Now the same two-moons network as the last lesson, written against the autograd engine. Compare this with the hand-derived backward pass — the forward pass is all that is left:

PythonPython not started

Five lines carry the whole loop, and they are the same five in every PyTorch program ever written:

logits = model(inputs)        forward
loss   = criterion(logits, y) measure
optimizer.zero_grad()         clear last step's gradients
loss.backward()               fill in this step's
optimizer.step()              move the parameters

The real thing

Here is the identical network in PyTorch. Run it on your own machine:

import torch
import torch.nn as nn
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split

X, y = make_moons(n_samples=600, noise=0.2, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0, stratify=y
)

torch.manual_seed(0)
inputs = torch.tensor(X_train, dtype=torch.float32)
targets = torch.tensor(y_train, dtype=torch.long)

model = nn.Sequential(
    nn.Linear(2, 16),
    nn.ReLU(),
    nn.Linear(16, 2),
)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.5)

for epoch in range(1, 401):
    logits = model(inputs)
    loss = criterion(logits, targets)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if epoch % 100 == 0:
        print(f"epoch {epoch:3}  loss {loss.item():.4f}")

model.eval()
with torch.no_grad():
    test_logits = model(torch.tensor(X_test, dtype=torch.float32))
    accuracy = (test_logits.argmax(dim=1).numpy() == y_test).mean()
print("test accuracy:", round(float(accuracy), 4))

Four things in that listing are worth naming:

  • nn.Module — a container that knows its parameters. nn.Sequential is the simplest one; writing your own means subclassing nn.Module and defining forward.
  • nn.CrossEntropyLoss expects raw logits, not probabilities. It applies the log-softmax internally, for exactly the numerical-stability reasons module 3 covered. Passing it a softmax output is a real and common bug.
  • optimizer.zero_grad() must come before backward(), because gradients accumulate.
  • torch.no_grad() switches off graph recording for evaluation, which saves memory and time.

A custom module, for when Sequential is not enough:

class MoonNet(nn.Module):
    def __init__(self, hidden: int = 16) -> None:
        super().__init__()
        self.first = nn.Linear(2, hidden)
        self.second = nn.Linear(hidden, 2)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.second(torch.relu(self.first(x)))

model = MoonNet()
print(sum(p.numel() for p in model.parameters()), "parameters")

And the parts you meet as soon as the data does not fit in one batch:

from torch.utils.data import TensorDataset, DataLoader

dataset = TensorDataset(inputs, targets)
loader = DataLoader(dataset, batch_size=32, shuffle=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

for epoch in range(20):
    for batch_inputs, batch_targets in loader:
        batch_inputs = batch_inputs.to(device)
        batch_targets = batch_targets.to(device)

        loss = criterion(model(batch_inputs), batch_targets)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

torch.save(model.state_dict(), "moonnet.pt")

Try it yourself

Exercise 1

Write matmul_backward(grad, A, B), returning the gradients with respect to A and B for C = A @ B.

Your turnPython not started
Exercise 2

Write sgd_step(parameters, learning_rate): move each parameter against its gradient, then zero that gradient — what optimizer.step() and zero_grad() do together.

Your turnPython not started

Check yourself

Why does PyTorch require optimizer.zero_grad() before loss.backward()?

Check yourself

You pass softmax probabilities to nn.CrossEntropyLoss and training goes badly. Why?

Recap

  • A tensor is an array that records the operations applied to it; that record is what backward() walks.
  • backward() topologically sorts the graph and visits it in reverse, so each node’s gradient is complete before it passes anything back.
  • Gradients accumulate, which is why zero_grad() exists and why forgetting it corrupts training silently.
  • Every PyTorch loop is forward, loss, zero_grad, backward, step — unchanged at any scale.
  • nn.CrossEntropyLoss takes raw logits and class indices; wrap evaluation in torch.no_grad().
  • The autograd on this page is not a simplification of the idea, only of the engineering.