Split documents so a retrieved passage retains the conditions needed to interpret it: the subject, applicable section, units, exceptions, and source identity. A chunk that fits an embedding model can still be too incomplete to support an answer. Choose boundaries against real questions, then inspect the evidence that survives retrieval.
A retention policy is a useful example. “Exports remain available for thirty days” may be a true sentence and a wrong answer to a trial user's question if the next paragraph sets a seven-day exception. The problem begins before generation: the retriever may never give the answering model the exception.
Start with the smallest complete piece of evidence
Microsoft's chunking guide describes fixed-size, variable-size, and combined approaches. Its examples are starting points for Azure search workloads, not a universal token target. The Architecture Center's chunking discussion emphasizes how document structure affects the choice and why very small chunks can lose necessary context.
For each document type, identify a useful evidence unit. In a procedure it may be a step with its prerequisite. In a table it may be a row with its column headings and unit. In a policy it may be a rule with its scope and exception. These units need not have equal lengths.
Keep the original document identity even when the text is transformed. A passage should lead back to a version and location that a reader can inspect. If a rule refers to another document, store the relationship and retrieve the referenced material when needed; do not silently splice two sources into an apparently continuous quotation.
A small corpus exposes two boundary failures
We created two synthetic documents. The first contains:
Standard exports
Exports remain available for thirty days after creation. Operators
can download the archive from the console during that period.
Exception: Trial exports are deleted after seven days, regardless
of the standard retention period.The second contains a small table:
Export limits
Plan | Maximum rows per export
Starter | 500
Team | 5000
The limits apply to one export, not the monthly total.The question “How long does a trial export remain available?” needs the exception and enough context to distinguish it from the standard rule. “How many rows can a Team export contain?” needs the row, the column meaning, and the per-export scope. A bare 5000 is not a complete fact.
The local fixture compared consecutive 12-word chunks without overlap against structure-preserving chunks that retained each complete miniature document. It checked whether all required evidence strings for each question occurred together in at least one chunk.
| Evidence requirement | Fixed 12-word split | Structure-preserving split |
|---|---|---|
| Standard retention plus trial exception | Absent from any single chunk | Present together |
| Table heading, Team row, and per-export scope | Absent from any single chunk | Present together |
Both assertions passed. This is a deliberately small boundary demonstration, not a retrieval benchmark. The structural chunks are larger, and the fixed width was chosen to expose a failure. No embeddings, ranker, or answering model ran. The result does not establish that larger chunks or structural splitting always perform better.
Preserve structure without returning whole manuals
For a large policy, keeping the entire document in every chunk would defeat selective retrieval. Instead, carry a concise section path and attach the applicable exception, or retrieve a parent section after finding a smaller child passage. The best implementation depends on whether exceptions are local, referenced elsewhere, or defined by a document-wide scope statement.
For tables, preserve column names and units beside each row. Repeating a short header can be cheaper to interpret than forcing the answerer to reconstruct it from a separately retrieved fragment. Keep footnotes that modify the value. Do not assume layout extraction captured merged cells, superscripts, or a scanned note correctly; inspect representative source pages before evaluating the splitter.
Overlap can protect text near a boundary, but it cannot reliably recover a distant exception. It also adds repeated material to an index and may crowd the top results with neighboring chunks from one document. Measure unique evidence coverage as well as raw hit count.
Avoid making a summary the only surviving source. A generated table explanation may omit a condition or introduce an inference. Preserve the original location and text needed to check it. Metadata helps identify a passage; it does not repair a sentence whose meaning was lost during extraction.
Compare strategies at the same retrieval budget
Build a small set of answerable questions with required passages marked in the source. Include a heading-dependent question, a table with a footnote, a cross-section exception, and a question that the corpus cannot answer. Freeze the document revisions and query set before comparing configurations.
First inspect the produced chunks. Then run each retrieval configuration with the same query set and a comparable amount of returned text. Top five chunks of 100 tokens and top five chunks of 1,000 tokens consume very different context budgets. Report both the result count and the text volume, using the actual tokenizer when token accounting matters.
For each question, trace whether the required evidence was extracted, indexed, retrieved, and retained in the final context. Only after that should you inspect whether the answer used it correctly. Context windows, retrieval, and memory explains those separate stages; chunking controls one part of the evidence path.
Choose the simplest strategy that preserves the necessary meaning at an acceptable retrieval cost. Keep the failure examples in the evaluation set. A future parser change can reintroduce the missing footnote even when the chunk-size setting stays unchanged.
Research and fixture date: October 28, 2026. This article reports deterministic string-boundary checks on synthetic text, not measured RAG answer quality.
