Skip to content

Retrieval and RAG·Lesson 54 of 67·expert

Building a vector index

Exact search is one matrix product and works further than you think. Build an index with metadata filtering and persistence, then learn what approximate search buys and what it costs.


In this lesson · 20 min

  • →Build an exact index with add, search and persistence
  • →Filter by metadata, and know why pre-filtering beats post-filtering
  • →Estimate when brute force stops being enough
  • →Explain what approximate search trades away
  • →Handle updates and deletes without corrupting the index

A vector index is an array of unit vectors, a parallel list of metadata, and a dot product. Start there — a great many production systems never need more.

PythonPython not started

A hundred thousand 768-dimension vectors is about 300 MB and searches in a few tens of milliseconds. That is a perfectly good production system for most internal corpora, and it has no recall loss, no index build step and no extra service to operate.

The index

PythonPython not started

The minimum argument is the similarity floor from the last lesson, and it is the difference between “here are five passages” and “here are the passages that are actually about this”.

Filtering by metadata

Real queries are rarely “search everything”. They are “search this customer’s documents, in English, published this year”.

PythonPython not started

Both return the same answers here, because exact search ranks everything anyway. The difference matters the moment you move to approximate search:

Persisting it

Vectors go in a binary file; metadata goes in JSONL. Both are boring, which is the point:

PythonPython not started

The manifest is what makes the model check from the last lesson enforceable across a restart.

Updates and deletes

The naive delete — removing a row from the array — invalidates every position after it. A tombstone keeps positions stable:

PythonPython not started

Masking with -inf rather than filtering keeps the index a single contiguous array, which is what makes the search fast.

When brute force is not enough

PythonPython not started

What approximate search trades

PythonPython not started

That is the entire bargain: search fewer candidates, miss some true neighbours. Probing every bucket recovers exact results and exact cost. Real indexes — HNSW, IVF-PQ — are far better at the trade than this toy, but the shape is identical, and recall is a number you must measure, not a property you can assume.

Try it yourself

Exercise 1

Write filtered_search(vectors, metadata, query, predicate, k): narrow to the rows the predicate accepts, then rank only those. Return (original_position, score) pairs, best first.

Your turnPython not started
Exercise 2

Write save_index and load_index: persist vectors, records and a manifest, and on load refuse a directory whose counts or dimensions disagree.

Your turnPython not started

Check yourself

Why is post-filtering an approximate index dangerous?

Check yourself

You have 60,000 chunks. What should you reach for?

Recap

  • An index is unit vectors, parallel metadata and a dot product. Exact search handles ~100k vectors comfortably.
  • Apply a similarity floor, or you always return k results however irrelevant.
  • Pre-filter, then search. Post-filtering an approximate index returns nothing when the filter is selective.
  • Persist vectors as .npy, records as JSONL, and a manifest carrying the model and dimensions so the compatibility check survives a restart.
  • Delete with a tombstone mask and compact offline; rebuilding the array per delete invalidates every position.
  • Approximate search trades recall for speed. Measure the recall — never assume it.