What is chunking?
Chunking is the process of splitting documents into smaller passages before they are embedded and indexed for retrieval. An embedding model represents a passage as a single vector, and a vector that stands for an entire fifty-page contract captures very little of any one clause. Splitting the contract into clause-sized chunks lets the search find the passage that answers a question and lets the system hand the model only the text it needs.
Chunk size is a tradeoff. Small chunks are precise but lose surrounding context, so a retrieved sentence may depend on a defined term explained two paragraphs earlier. Large chunks keep context but dilute the embedding and consume more of the model’s context window. Most pipelines settle between a few hundred and a thousand tokens per chunk, with some overlap between neighbors so that sentences at a boundary appear in both.
Common chunking techniques
- Fixed-size chunking cuts text every set number of tokens with an overlap. It is simple and predictable but ignores document structure.
- Structure-aware chunking splits on headings, paragraphs, list items, and table boundaries, so each chunk matches a unit the author intended.
- Semantic chunking uses embeddings to detect where the topic shifts and splits there, which suits long unstructured text such as meeting transcripts.
- Parent-child chunking indexes small chunks for precise matching but returns the larger parent section to the model, giving both precision and context.
- Metadata enrichment prepends the document title, section path, and effective date to each chunk so a retrieved passage carries its own provenance.
Chunking in practice
A law firm indexes its library of negotiated master services agreements so associates can see how particular clauses were handled in past deals. Fixed-size chunks split indemnification clauses mid-sentence and separated defined terms from their definitions, so retrieved passages were relevant in theory and useless in practice. Switching to structure-aware chunking at the clause level, with the agreement name and section heading prepended to each chunk, made the passages readable on their own and raised the share of test questions answered correctly.
What to watch for
- Tables and spreadsheets need their own handling, because splitting a table by row count separates values from their column headers.
- Test several chunk sizes against real questions rather than accepting a default, since the best size depends on the documents and the questions.
- Scanned documents go through optical character recognition before chunking, and OCR errors propagate into every downstream step.
- Changing the chunking strategy means re-indexing the entire corpus, so decide early and record the choice.