Skip to content

Retrieval and RAG·Lesson 52 of 67·expert

Embeddings and vector similarity

Turn text into vectors whose geometry means something, measure similarity properly, and handle the practical parts — batching, caching and the fact that you can never mix two models' vectors.


In this lesson · 20 min

  • →Explain what an embedding is and what its geometry encodes
  • →Compute cosine similarity over a batch, without loops
  • →Rank documents against a query and take the top k
  • →Batch and cache embedding calls
  • →Explain why two models' embeddings can never be compared

What an embedding is

A function from text to a fixed-length vector of floats, trained so that texts with similar meaning land close together.

PythonPython not started

Every document becomes a point in the same space. Nothing about a single vector is interpretable — dimension 7 means nothing on its own. What means something is distance between vectors.

Cosine similarity

Module 3 built this: normalise, then take the dot product.

PythonPython not started

Read the diagonal — every document is identical to itself at 1.0 — and then look at the off-diagonal entries. The two RAG sentences share vocabulary and score highest; the cat is orthogonal to everything.

Ranking against a query

PythonPython not started

That is a search engine: one matrix-vector product and an argpartition, exactly the top-k pattern from module 3.

The real call

With a hosted embedding model the shape is the same; only embed changes:

# An embedding model is a separate service from the chat model. Anthropic
# recommends a dedicated provider such as Voyage AI; the shape below is
# typical of all of them.
def embed(texts: list[str]) -> np.ndarray:
    response = embeddings_client.embed(
        model="voyage-3",
        input=texts,             # a batch, not one string at a time
        input_type="document",   # or "query" — many models distinguish the two
    )
    vectors = np.array([item.embedding for item in response.data], dtype=np.float32)
    return vectors / np.linalg.norm(vectors, axis=1, keepdims=True)

Two details worth carrying over:

  • Batch. One request with 128 texts is far cheaper and faster than 128 requests. Reach for itertools.batched from module 2.
  • Ask for the right input_type. Several models embed a question and a passage differently, because a question is not a shorter version of its answer.
PythonPython not started

Caching

Embedding the same text twice is money spent on an answer you already had. Key a cache by a hash of the content, exactly as in module 2:

PythonPython not started

Note the model name in the cache key. That is not decoration.

You cannot mix models

Two models’ vectors live in unrelated spaces. Comparing them produces a number, and the number is meaningless:

PythonPython not started
PythonPython not started

Dimensions and memory

PythonPython not started

Smaller embeddings are cheaper to store and faster to search, and lose some accuracy. Many models now support shortening a vector deliberately rather than training a smaller one — measure the retrieval quality you actually lose before assuming it matters.

Try it yourself

Exercise 1

Write top_k(index, query_vector, k): return a list of (position, score) pairs for the k closest rows, best first. The index rows are already unit vectors.

Your turnPython not started
Exercise 2

Give VectorStore an add and a search that both refuse a mismatched model or dimension, raising ValueError.

Your turnPython not started

Check yourself

You upgrade to a newer embedding model but keep the existing index. What happens?

Check yourself

Why normalise embeddings when you store them rather than at query time?

Recap

  • An embedding maps text to a fixed-length vector; only distances between vectors mean anything.
  • Normalise at index time, then similarity is a dot product and top-k is argpartition.
  • Batch embedding calls and cache them by a hash of the content plus the model id.
  • Ask for the right input_type when a model distinguishes queries from documents.
  • Two models’ vectors are incomparable. Store the model and dimension with the index and refuse a mismatch, because nothing else will tell you.
  • Retrieval always returns k results; a similarity floor is what stops irrelevant context reaching the model.