Skip to content

Numerical Python·Lesson 24 of 67·intermediate

Linear algebra you actually need

Dot products, matrix multiplication, norms and cosine similarity — the handful of operations that every embedding, attention head and linear layer is built from.


In this lesson · 20 min

  • →Compute dot products and read what they mean geometrically
  • →Multiply matrices with @, and check the shapes line up
  • →Measure length with a norm, and normalise to unit vectors
  • →Implement cosine similarity for one query and for a whole batch
  • →Recognise a linear layer, and solve a least-squares fit

You do not need a linear algebra course to do AI engineering. You need four operations, and a firm grip on what the shapes mean.

The dot product

Multiply matching elements, add the results:

PythonPython not started

Geometrically it measures agreement: large and positive when two vectors point the same way, zero when they are perpendicular, negative when they oppose:

PythonPython not started

Note that also_same scores 3, not 1, purely because it is longer. That is why raw dot products are a poor similarity measure, and why cosine similarity divides the length out.

Norms

The norm is a vector’s length:

PythonPython not started

Cosine similarity

Dot product divided by both lengths — equivalently, the dot product of the unit vectors. It ranges from -1 to 1 and ignores magnitude entirely:

PythonPython not started

For a whole corpus, normalise once and the similarity is a single matrix-vector product:

PythonPython not started

Matrix multiplication

A @ B requires the inner dimensions to match: (n, k) @ (k, m) gives (n, m). Reading the shapes is most of the skill:

PythonPython not started
PythonPython not started

The fix is nearly always a transpose. A @ C.T turns (2, 3) @ (3, 2) into (2, 2):

PythonPython not started

A linear layer

A neural network layer is one matrix multiplication and one broadcast addition:

PythonPython not started

Stack two of those with an activation between them and you have a neural network — which is exactly what you will build by hand in module 5.

Other operations worth knowing

PythonPython not started

Least squares

Fitting a straight line is solving an overdetermined system, and NumPy does it directly. This is linear regression, before scikit-learn ever gets involved:

PythonPython not started

Try it yourself

Exercise 1

Write cosine_similarity(query, documents) returning one score per document row, then set best to the index of the closest document.

Your turnPython not started
Exercise 2

Complete the linear layer: compute outputs as inputs times weights plus bias, then apply ReLU into activated.

Your turnPython not started

Check yourself

A is (32, 768) and B is (768, 4). What is A @ B, and what would A * B do?

Check yourself

Why do vector databases store normalised embeddings and then use a plain dot product?

Recap

  • The dot product measures agreement, but it grows with length — which is why raw dot products make a poor similarity.
  • np.linalg.norm(x, axis=1, keepdims=True) gives per-row lengths ready to divide by.
  • Cosine similarity is the dot product of unit vectors, from -1 to 1, independent of magnitude.
  • @ is matrix multiplication and needs the inner dimensions to match; * is elementwise. A transpose is usually the fix.
  • A linear layer is inputs @ weights + bias, with the bias broadcast across the batch.
  • Solve systems with np.linalg.solve or np.linalg.lstsq, never by inverting a matrix.