Chunking Strategies for Production RAG

There is no best chunk size for RAG. A 400-token chunk can preserve a support article and destroy a table, a contract clause, or a function definition.

Choose boundaries from the structure of the source and the questions users ask. Then measure whether the retrieved unit contains enough evidence to answer without dragging unrelated text into the prompt.

Define the job of a chunk

A chunk serves two different systems:

1. The retriever needs a focused representation that matches a query.
2. The generator needs enough context to interpret the matched passage.

One unit rarely optimizes both. That is why parent-child retrieval is useful: embed and match a small child passage, then return its larger parent section to the model.

Before tuning chunk size, establish the retrieval metrics in RAG Evaluation: Metrics for Retrieval and Answer Quality. Otherwise “the answers feel better” becomes the experiment.

Start with document structure

Do not flatten the document to plain text and hope a recursive character splitter reconstructs its meaning. Parse headings, lists, tables, code fences, and page metadata before splitting.

Use size as a constraint, not the strategy

Token limits still matter. Embedding models accept bounded inputs, and huge chunks produce diluted representations. But a token target should decide when a structural unit must be subdivided—not where every document is cut.

A workable starting experiment is:

A: structural sections, capped at 300 tokens
B: structural sections, capped at 600 tokens
C: 250-token children retrieving 800-token parents

Run the same query set against all three. Compare recall@K, context precision, and answer correctness. Segment results by question type; definitions and multi-step procedures often prefer different units.

Overlap fixes one problem and creates another

Overlap helps when a fact crosses an arbitrary boundary. It also duplicates evidence, inflates the index, and can fill the top results with near-identical passages.

Prefer semantic continuity first:

• repeat the heading path in metadata or chunk text;
• keep list introductions with their items;
• keep table headers with every row group;
• attach definitions to the section that uses them;
• use adjacent-chunk expansion after retrieval.

Add fixed overlap only where boundary tests show missed evidence. Measure duplicate-result rate alongside recall.

Store provenance that survives retrieval

Every chunk should carry enough metadata to explain and filter it:

{
  "chunk_id": "policy-v7:4.2:01",
  "document_id": "access-policy",
  "document_version": 7,
  "heading_path": ["Contractors", "Production access"],
  "tenant_id": "acme",
  "effective_at": "2026-07-01",
  "source_uri": "/policies/access#contractors"
}

Versioning matters. If old and new policy chunks coexist without freshness rules, excellent semantic retrieval can return the wrong policy confidently.

Build a boundary test set

Include questions whose answers sit:

• at the start and end of sections;
• across two adjacent sections;
• inside a table with units in the header;
• in a list whose meaning depends on its introduction;
• under duplicate headings on different pages;
• in a newer version that contradicts an older one.

Also include no-answer questions. Large chunks sometimes make an unrelated passage look plausible enough for the generator to answer anyway.

Diagnose chunking failures

The exact-ID failure belongs to retrieval. Hybrid Search vs Vector Search for RAG explains why changing chunk size will not fix every miss.

Chunking experiment checklist

• Split on source structure before applying token caps.
• Preserve heading, version, tenant, and source metadata.
• Compare at least three chunking configurations on one fixed dataset.
• Measure retrieval and answer quality separately.
• Track duplicate-result rate when using overlap.
• Test tables, lists, code, version conflicts, and boundary-spanning answers.
• Re-index into a versioned index so rollback is possible.

Sources

• Microsoft: Chunk large documents for RAG and vector search
• Microsoft: RAG content preparation and retrieval guidance
• OpenAI embeddings guide