Numerical Python·Lesson 24 of 67·intermediate
Linear algebra you actually need
Dot products, matrix multiplication, norms and cosine similarity — the handful of operations that every embedding, attention head and linear layer is built from.
In this lesson · 20 min
- →Compute dot products and read what they mean geometrically
- →Multiply matrices with @, and check the shapes line up
- →Measure length with a norm, and normalise to unit vectors
- →Implement cosine similarity for one query and for a whole batch
- →Recognise a linear layer, and solve a least-squares fit
You do not need a linear algebra course to do AI engineering. You need four operations, and a firm grip on what the shapes mean.
The dot product
Multiply matching elements, add the results:
Geometrically it measures agreement: large and positive when two vectors point the same way, zero when they are perpendicular, negative when they oppose:
Note that also_same scores 3, not 1, purely because it is longer. That is why raw dot products
are a poor similarity measure, and why cosine similarity divides the length out.
Norms
The norm is a vector’s length:
Cosine similarity
Dot product divided by both lengths — equivalently, the dot product of the unit vectors. It ranges from -1 to 1 and ignores magnitude entirely:
For a whole corpus, normalise once and the similarity is a single matrix-vector product:
Matrix multiplication
A @ B requires the inner dimensions to match: (n, k) @ (k, m) gives (n, m). Reading the
shapes is most of the skill:
The fix is nearly always a transpose. A @ C.T turns (2, 3) @ (3, 2) into (2, 2):
A linear layer
A neural network layer is one matrix multiplication and one broadcast addition:
Stack two of those with an activation between them and you have a neural network — which is exactly what you will build by hand in module 5.
Other operations worth knowing
Least squares
Fitting a straight line is solving an overdetermined system, and NumPy does it directly. This is linear regression, before scikit-learn ever gets involved:
Try it yourself
Write cosine_similarity(query, documents) returning one score per document row, then set best to the index of the closest document.
Complete the linear layer: compute outputs as inputs times weights plus bias, then apply ReLU into activated.
Check yourself
A is (32, 768) and B is (768, 4). What is A @ B, and what would A * B do?
Check yourself
Why do vector databases store normalised embeddings and then use a plain dot product?
Recap
- The dot product measures agreement, but it grows with length — which is why raw dot products make a poor similarity.
np.linalg.norm(x, axis=1, keepdims=True)gives per-row lengths ready to divide by.- Cosine similarity is the dot product of unit vectors, from -1 to 1, independent of magnitude.
@is matrix multiplication and needs the inner dimensions to match;*is elementwise. A transpose is usually the fix.- A linear layer is
inputs @ weights + bias, with the bias broadcast across the batch. - Solve systems with
np.linalg.solveornp.linalg.lstsq, never by inverting a matrix.