Skip to content

Numerical Python·Lesson 22 of 67·intermediate

Broadcasting rules

How NumPy stretches a small array to meet a large one, the two-line rule that decides whether it will, and the keepdims trick that makes row-normalising a batch of embeddings a one-liner.


In this lesson · 18 min

  • →State the broadcasting rule and apply it to any pair of shapes
  • →Predict the result shape before running the code
  • →Use keepdims so a reduction still broadcasts back
  • →Add a length-1 axis deliberately to control how shapes meet
  • →Read and fix a broadcasting error message

Broadcasting is how NumPy makes arrays of different shapes work together without copying data or writing loops. It is the single most useful idea in the library, and the one that produces the most confusing error messages until you know the rule.

The simplest case

PythonPython not started

Nothing was copied and no loop ran: the scalar was broadcast across the array.

The rule

Line the shapes up from the right. Two dimensions are compatible when they are equal, or when one of them is 1. A missing dimension on the left counts as 1.

      batch  (32, 768)
      bias   (    768,)   ->  treated as (1, 768)
      result (32, 768)    ->  the bias is reused for every row

      batch  (32, 768)
      norms  (32,   1)
      result (32, 768)    ->  each row divided by its own number

      a      (5, 1)
      b      (1, 3)
      result (5, 3)       ->  both are stretched
PythonPython not started

The bias has three numbers and the batch has three columns, so the bias is applied to every row. This is exactly what adding a bias vector to a layer’s output does.

PythonPython not started

Both were stretched: the column across four columns, the row down three rows. That gives you every combination — an outer operation, with no loop.

When it fails

PythonPython not started

Read the message right to left: it names the axis where the sizes disagree. The fix is always either to transpose one array, or to add a length-1 axis so the sizes line up as you intended.

PythonPython not started

keepdims

A reduction removes the axis it reduced, which is usually what you want — until you need the result to broadcast back against the original:

PythonPython not started

Without keepdims, batch / totals would line (2, 3) up against (2,) and fail — or worse, succeed for the wrong shape and silently divide the wrong way.

Normalising a batch of embeddings

This is the broadcasting expression you will write most often in AI work:

PythonPython not started

Centring and standardising

PythonPython not started

Here the statistic is per column, with shape (2,), and NumPy’s right-alignment does the right thing with no newaxis at all. Per-column needs nothing; per-row needs keepdims.

Similarity without a loop

One query against every document, in a single expression:

PythonPython not started

And every pair of documents against each other, which is how de-duplication works:

PythonPython not started

documents[:, np.newaxis, :] has shape (3, 1, 3) and documents[np.newaxis, :, :] has shape (1, 3, 3). Broadcasting stretches both to (3, 3, 3) — every pair, without a loop.

Try it yourself

Exercise 1

Set unit to embeddings with every row scaled to length 1, using one broadcasting expression.

Your turnPython not started
Exercise 2

Build table: a (4, 3) array where table[i, j] is lengths[i] * weights[j]. Use broadcasting, not a loop.

Your turnPython not started

Check yourself

An array of shape (32, 768) is added to one of shape (768,). What is the result shape?

Check yourself

Why does dividing a (2, 3) batch by batch.sum(axis=1) fail, while keepdims=True works?

Recap

  • Align shapes from the right. Dimensions are compatible when equal or when one is 1, and a missing leading dimension counts as 1.
  • A (n,) array lines up with the columns. For a per-row value use [:, np.newaxis] or keepdims=True.
  • keepdims=True is what lets a reduction broadcast back against the array it came from.
  • embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True) normalises a batch — the expression behind every cosine similarity.
  • Per-column statistics broadcast with no help; per-row statistics always need the extra axis.
  • Broadcasting hides allocations: check the result shape before running it on real data.