Retrieval and RAG·Lesson 53 of 67·expert
Chunking strategies that survive contact with real documents
Splitting on a character count cuts sentences in half and answers with them. Split on structure, overlap deliberately, and carry the metadata that lets a chunk explain where it came from.
In this lesson · 20 min
- →Explain why documents are chunked at all, and what a chunk competes on
- →Implement fixed-size chunking with overlap
- →Split on structure, falling back progressively
- →Attach the metadata a chunk needs to be useful later
- →Choose a chunk size from what you are retrieving for
Chunking sets the ceiling on everything downstream. A passage that answers the question but was split across two chunks cannot be retrieved as an answer by any ranking function, however good.
Why chunk
Fixed size, and why it fails
The simplest splitter cuts every N characters. Watch what it does to the prose:
Chunk boundaries land mid-word and mid-sentence. A query about overlap now has to match a chunk that begins “n is not cut in half” — and the chunk that contains the actual advice has lost the word “overlap” to its neighbour.
Overlap
Repeating a slice of each chunk in the next one means a sentence cut at a boundary still appears whole somewhere:
Overlap buys robustness and costs storage — every chunk is embedded, and the duplicated words are embedded twice. Ten to twenty per cent is the usual compromise.
Split on structure
A document already tells you where its boundaries are. Use them before falling back to counting:
Markdown headings are better still, because they carry the section a passage belongs to:
Recursive splitting
The practical strategy: try the biggest natural boundary; if a piece is still too large, split it on the next boundary down; only count characters as a last resort.
Every chunk now ends on a sentence or a paragraph, and none exceeds the limit.
Carry the metadata
A chunk with no provenance cannot be cited, filtered or de-duplicated:
Choosing a size
That last rule is worth emphasising: retrieve at 200 tokens for precision, then send the parent section so the model has enough to work with. The two jobs have different optimal sizes.
Try it yourself
Write chunk_words(text, size, overlap): split into chunks of size words that share overlap words with the previous chunk, and raise ValueError if the overlap is not smaller than the size.
Write split_sections(markdown): return (heading, body) pairs, where heading is the nearest preceding ATX heading text, or None for text before the first one. Skip sections with an empty body.
Check yourself
An answer exists in your corpus but retrieval never finds it. What should you check first?
Check yourself
Why retrieve small chunks but send the surrounding section to the model?
Recap
- Chunking sets the ceiling: information split across a boundary cannot be retrieved whole.
- Fixed-size splitting cuts words and sentences; overlap of 10-20% repairs most of the damage at the cost of duplicated storage.
- Guard the overlap against being larger than the size, or the loop never ends.
- Split on structure first — paragraphs, then headings — and fall back progressively, counting characters last.
- Carry
doc_id, heading, position and a content hash on every chunk, and consider prefixing the text with its heading path. - Retrieve small for precision, then expand to the parent section before answering.