Retrieval and RAG·Lesson 52 of 67·expert
Embeddings and vector similarity
Turn text into vectors whose geometry means something, measure similarity properly, and handle the practical parts — batching, caching and the fact that you can never mix two models' vectors.
In this lesson · 20 min
- →Explain what an embedding is and what its geometry encodes
- →Compute cosine similarity over a batch, without loops
- →Rank documents against a query and take the top k
- →Batch and cache embedding calls
- →Explain why two models' embeddings can never be compared
What an embedding is
A function from text to a fixed-length vector of floats, trained so that texts with similar meaning land close together.
Every document becomes a point in the same space. Nothing about a single vector is interpretable — dimension 7 means nothing on its own. What means something is distance between vectors.
Cosine similarity
Module 3 built this: normalise, then take the dot product.
Read the diagonal — every document is identical to itself at 1.0 — and then look at the off-diagonal entries. The two RAG sentences share vocabulary and score highest; the cat is orthogonal to everything.
Ranking against a query
That is a search engine: one matrix-vector product and an argpartition, exactly the top-k
pattern from module 3.
The real call
With a hosted embedding model the shape is the same; only embed changes:
# An embedding model is a separate service from the chat model. Anthropic
# recommends a dedicated provider such as Voyage AI; the shape below is
# typical of all of them.
def embed(texts: list[str]) -> np.ndarray:
response = embeddings_client.embed(
model="voyage-3",
input=texts, # a batch, not one string at a time
input_type="document", # or "query" — many models distinguish the two
)
vectors = np.array([item.embedding for item in response.data], dtype=np.float32)
return vectors / np.linalg.norm(vectors, axis=1, keepdims=True)
Two details worth carrying over:
- Batch. One request with 128 texts is far cheaper and faster than 128 requests. Reach for
itertools.batchedfrom module 2. - Ask for the right
input_type. Several models embed a question and a passage differently, because a question is not a shorter version of its answer.
Caching
Embedding the same text twice is money spent on an answer you already had. Key a cache by a hash of the content, exactly as in module 2:
Note the model name in the cache key. That is not decoration.
You cannot mix models
Two models’ vectors live in unrelated spaces. Comparing them produces a number, and the number is meaningless:
Dimensions and memory
Smaller embeddings are cheaper to store and faster to search, and lose some accuracy. Many models now support shortening a vector deliberately rather than training a smaller one — measure the retrieval quality you actually lose before assuming it matters.
Try it yourself
Write top_k(index, query_vector, k): return a list of (position, score) pairs for the k closest rows, best first. The index rows are already unit vectors.
Give VectorStore an add and a search that both refuse a mismatched model or dimension, raising ValueError.
Check yourself
You upgrade to a newer embedding model but keep the existing index. What happens?
Check yourself
Why normalise embeddings when you store them rather than at query time?
Recap
- An embedding maps text to a fixed-length vector; only distances between vectors mean anything.
- Normalise at index time, then similarity is a dot product and top-k is
argpartition. - Batch embedding calls and cache them by a hash of the content plus the model id.
- Ask for the right
input_typewhen a model distinguishes queries from documents. - Two models’ vectors are incomparable. Store the model and dimension with the index and refuse a mismatch, because nothing else will tell you.
- Retrieval always returns k results; a similarity floor is what stops irrelevant context reaching the model.