Skip to content

Retrieval and RAG·Lesson 53 of 67·expert

Chunking strategies that survive contact with real documents

Splitting on a character count cuts sentences in half and answers with them. Split on structure, overlap deliberately, and carry the metadata that lets a chunk explain where it came from.


In this lesson · 20 min

  • →Explain why documents are chunked at all, and what a chunk competes on
  • →Implement fixed-size chunking with overlap
  • →Split on structure, falling back progressively
  • →Attach the metadata a chunk needs to be useful later
  • →Choose a chunk size from what you are retrieving for

Chunking sets the ceiling on everything downstream. A passage that answers the question but was split across two chunks cannot be retrieved as an answer by any ranking function, however good.

Why chunk

PythonPython not started

Fixed size, and why it fails

The simplest splitter cuts every N characters. Watch what it does to the prose:

PythonPython not started

Chunk boundaries land mid-word and mid-sentence. A query about overlap now has to match a chunk that begins “n is not cut in half” — and the chunk that contains the actual advice has lost the word “overlap” to its neighbour.

Overlap

Repeating a slice of each chunk in the next one means a sentence cut at a boundary still appears whole somewhere:

PythonPython not started

Overlap buys robustness and costs storage — every chunk is embedded, and the duplicated words are embedded twice. Ten to twenty per cent is the usual compromise.

Split on structure

A document already tells you where its boundaries are. Use them before falling back to counting:

PythonPython not started

Markdown headings are better still, because they carry the section a passage belongs to:

PythonPython not started

Recursive splitting

The practical strategy: try the biggest natural boundary; if a piece is still too large, split it on the next boundary down; only count characters as a last resort.

PythonPython not started

Every chunk now ends on a sentence or a paragraph, and none exceeds the limit.

Carry the metadata

A chunk with no provenance cannot be cited, filtered or de-duplicated:

PythonPython not started

Choosing a size

PythonPython not started

That last rule is worth emphasising: retrieve at 200 tokens for precision, then send the parent section so the model has enough to work with. The two jobs have different optimal sizes.

Try it yourself

Exercise 1

Write chunk_words(text, size, overlap): split into chunks of size words that share overlap words with the previous chunk, and raise ValueError if the overlap is not smaller than the size.

Your turnPython not started
Exercise 2

Write split_sections(markdown): return (heading, body) pairs, where heading is the nearest preceding ATX heading text, or None for text before the first one. Skip sections with an empty body.

Your turnPython not started

Check yourself

An answer exists in your corpus but retrieval never finds it. What should you check first?

Check yourself

Why retrieve small chunks but send the surrounding section to the model?

Recap

  • Chunking sets the ceiling: information split across a boundary cannot be retrieved whole.
  • Fixed-size splitting cuts words and sentences; overlap of 10-20% repairs most of the damage at the cost of duplicated storage.
  • Guard the overlap against being larger than the size, or the loop never ends.
  • Split on structure first — paragraphs, then headings — and fall back progressively, counting characters last.
  • Carry doc_id, heading, position and a content hash on every chunk, and consider prefixing the text with its heading path.
  • Retrieve small for precision, then expand to the parent section before answering.