When a RAG answer is wrong, changing the prompt is usually the fastest way to hide the real failure.
Debug the pipeline in order. First prove that the correct source exists and is permitted. Then prove it was parsed, indexed, retrieved, and delivered to the model. Only then inspect generation.
The decision tree
Wrong or missing answer├─ Does the corpus contain current, authoritative evidence?│ ├─ No → content/freshness failure│ └─ Yes├─ Is that evidence present in the index and permitted for this user?│ ├─ No → ingestion, metadata, or authorization failure│ └─ Yes├─ Is it in the retrieved candidate set?│ ├─ No → query, embedding, keyword, filter, or chunking failure│ └─ Yes├─ Is it in the final prompt context?│ ├─ No → reranking, truncation, diversity, or assembly failure│ └─ Yes├─ Does the answer use it correctly?│ ├─ No → generation, instruction, or evidence-conflict failure│ └─ Yes└─ Is the displayed answer/citation still wrong?└─ formatting, citation mapping, cache, or UI failure
Save this sequence in the incident runbook. It prevents three engineers from tuning three different layers at once.
1. Content and freshness failures
The system cannot retrieve information it does not have. Confirm the authoritative source, effective date, and version before examining scores.
Typical failures:
- the source never entered the ingestion scope;
- a parser silently skipped a page or table;
- the new version was indexed but the old version was not retired;
- a cache serves an answer created before the update;
- two sources disagree and neither has an authority rank.
Record document_id, version, effective_at, indexed_at, and a content hash. A URL alone is not a version identifier.
2. Ingestion and representation failures
Inspect the exact indexed text, not the source PDF. OCR may have dropped columns. HTML extraction may have repeated navigation. A table may have become a stream of numbers without headers.
Create deterministic ingestion checks:
- expected document count by connector- percentage of empty or tiny chunks- parser errors and skipped pages- duplicate content hashes- chunks missing tenant, version, or source metadata- sample diff between extracted text and source
If the evidence is split badly, use the experiments in Chunking Strategies for Production RAG.
3. Retrieval failures
Search for the known relevant chunk using the exact production query, filters, index version, and embedding model. Developer-console searches often omit the permission filter or query rewrite that caused the incident.
| Symptom | First check |
|---|---|
| Identifier missing | keyword or hybrid retrieval |
| Conceptual match missing | embedding and query formulation |
| Correct tenant, wrong document | metadata and authority boost |
| Only three results after top-K 20 | restrictive filter behavior |
| Relevant chunk at rank 30 | candidate depth and reranking |
| Near-duplicates fill results | overlap and diversity rules |
Measure the retrieval layer separately with the RAG evaluation scorecard. Final-answer accuracy is too far downstream to isolate it.
4. Context assembly failures
A relevant candidate can disappear before inference. Common causes include:
- reranking pushed it below K;
- a token budget truncated the tail;
- deduplication removed the wrong copy;
- a template omitted metadata needed for interpretation;
- context ordering separated a table from its header;
- tool output was stored but never inserted into the prompt.
Log candidate IDs before and after every transformation. The LLM observability trace schema should make this comparison possible for one production request.
5. Generation failures
Once the correct evidence is visible in the final prompt, evaluate the answer against that prompt. Look for:
- instructions that reward completeness over abstention;
- conflicting evidence without dates or authority labels;
- a question requiring synthesis across omitted chunks;
- output schemas that discard qualifications;
- citations attached after generation by fuzzy text matching;
- prior conversation overriding current evidence.
Do not assume a larger model is the first fix. Make the evidence unambiguous, specify the refusal behavior, and test the changed case against the regression set.
6. Authorization failures are security incidents
If the answer uses evidence the user should not access, stop treating the event as a relevance bug. Preserve the trace, identify every exposed document and affected tenant, close the query path, and follow the incident process.
Filters must derive from authenticated application state. Never ask the model to supply or preserve the tenant ID.
The minimum incident record
Capture:
{"trace_id": "tr_291","question": "redacted or access-controlled","user_scope": ["tenant:acme", "group:support"],"index_version": "kb-2026-08-20","query_variants": ["..."],"candidate_chunk_ids": ["..."],"final_context_ids": ["..."],"prompt_version": "answer-v12","model": "recorded-deployment-id","failure_layer": "retrieval"}
Turn the incident into a labeled evaluation case before closing it.
Debugging checklist
- Verify authoritative evidence and its version.
- Inspect the exact indexed representation.
- Replay production query, filters, and versions.
- Compare candidates with final prompt context.
- Evaluate generation only after evidence is confirmed.
- Escalate unauthorized retrieval as a security incident.
- Add the failure to the regression dataset.
