Production RAG Architecture: The Complete Pipeline

Production RAG is two connected systems, not one vector-search call. The knowledge plane turns source material into versioned, permission-aware evidence units. The serving plane authenticates the user, retrieves and reranks allowed evidence, assembles context, generates an answer, maps citations, and records enough trace data to evaluate or debug the result.

The operating rule is: every stage emits a versioned artifact with a measurable contract. If a wrong answer cannot be localized to the corpus, ingestion, retrieval, context, generation, citation, authorization, or serving layer, the architecture is not observable enough to operate.

Start with two planes

KNOWLEDGE PLANE
sources → parse → normalize → segment → enrich → authorize → embed → index
   │         │          │         │          │          │          │
source_id  parser_v  content_v  chunk_v  metadata_v  policy_v  index_v

SERVING PLANE
identity → query → retrieve + filter → rerank → assemble → generate → cite
    │         │            │             │          │          │       │
 principal  query_v   candidate_set   ranking_v  context_v  model_v  evidence_ids
                                   ↓
                         trace, evaluation, feedback

The separation matters. Knowledge updates and user requests have different latency, scaling, and failure modes. A source deletion may require asynchronous re-indexing, while an online request must fail safely within seconds. They meet at a versioned index and an authorization contract.

If the use case does not require proprietary, current, or source-grounded evidence, first compare RAG with fine-tuning and long context. Retrieval adds an operating system that must earn its complexity.

The foundational RAG paper combined parametric generation with retrieved non-parametric memory and highlighted provenance and knowledge updates as important properties. Production applications generalize that idea beyond the paper's specific model and Wikipedia index. See Lewis et al., 2020.

Build a source registry before an index

The source registry answers questions a vector store cannot:

{
  "source_id": "policy_travel_2026",
  "canonical_uri": "https://intranet/policies/travel",
  "authority": "people-operations",
  "effective_at": "2026-07-01T00:00:00Z",
  "expires_at": null,
  "classification": "internal",
  "allowed_groups": ["employees"],
  "content_hash": "sha256:...",
  "parser_version": "html-v4",
  "status": "active"
}

Store ownership, freshness, classification, permission source, checksum, and lifecycle status before generating embeddings. This makes reprocessing, deletion, rollback, and audit possible.

Treat ingestion as a state machine: discovered → fetched → parsed → validated → indexed, with quarantined, superseded, and deleted as explicit states. A parser failure should not silently leave an old document looking current.

Preserve structure and provenance during ingestion

Parsing should retain headings, tables, lists, code blocks, page numbers, timestamps, and document relationships. Flattening everything into text destroys boundaries that retrieval and citation later need.

Create a stable evidence identity for every retrievable unit:

{
  "evidence_id": "policy_travel_2026#expenses.receipts:p3",
  "source_id": "policy_travel_2026",
  "heading_path": ["Expenses", "Receipts"],
  "page": 3,
  "text": "Receipts are required for expenses above EUR 25...",
  "content_version": "2026-07-01",
  "classification": "internal",
  "allowed_groups": ["employees"],
  "valid_from": "2026-07-01T00:00:00Z"
}

Chunk boundaries should follow source structure and expected questions, then be tested. A universal token size cannot preserve every clause, table, or function. The production chunking guide provides the boundary experiments and parent-child pattern.

Make authorization part of retrieval

Derive tenant, user, group, and classification scope from authenticated application identity. The model may rewrite search terms; it must not choose the security scope.

authenticated principal + policy context + query
                       ↓
permission-filtered candidate retrieval

Apply the filter inside the retrieval request or at a stronger index/infrastructure boundary. Retrieving another tenant's chunk and asking the model to ignore it is already a data-isolation failure. Cache keys, reranking batches, trace access, and evaluation datasets must preserve the same scope. The multi-tenant RAG isolation architecture details those controls and adversarial tests.

Retrieve candidates for the query population you have

Vector search handles semantic similarity; keyword search remains valuable for product codes, error strings, dates, names, and rare terms. Production traffic usually mixes both.

A strong baseline is permission-filtered hybrid retrieval followed by rank fusion. Measure candidate recall before adding complexity. The hybrid versus vector retrieval guide explains why raw BM25 and cosine scores should not be added without calibration.

Log the query representation, filters, retriever versions, candidate IDs, ranks, and scores. Without the complete candidate set, an investigator cannot distinguish “evidence was never retrieved” from “the reranker dropped it.”

Rerank only after candidate recall works

The first retriever searches a large corpus cheaply. A reranker can spend more computation on perhaps 20–100 candidates, but it cannot recover evidence absent from that set.

Use a labeled evaluation set to measure candidate recall at the handoff. Add reranking when required evidence is commonly present but ordered too low or surrounded by noise. Tune candidate depth, final context count, latency, and cost together. The RAG reranking guide provides the release rule and failure diagnosis.

Preserve document diversity and relationships. Six adjacent chunks from one page can crowd out a second source needed to resolve a conflict.

Assemble an evidence packet, not a text dump

The context assembler owns ordering, deduplication, token allocation, conflict presentation, and the final evidence manifest. Its output should be inspectable:

{
  "context_version": "assembler-v7",
  "query": "When is a receipt required?",
  "evidence": [
    {
      "evidence_id": "policy_travel_2026#expenses.receipts:p3",
      "source_title": "Travel Policy",
      "effective_at": "2026-07-01",
      "text": "Receipts are required for expenses above EUR 25..."
    }
  ],
  "excluded": [
    {"evidence_id": "policy_travel_2025#receipts:p2", "reason": "superseded"}
  ]
}

Do not make the model infer which version is current from two contradictory chunks. Encode authority and effective dates, prefer the applicable source, and expose unresolved conflicts when the business rule cannot choose automatically.

Generate with bounded evidence behavior

The generation contract should define:

• whether the model may use knowledge outside the supplied evidence;
• when it must abstain or ask a clarification;
• the required answer and citation structure;
• how to handle conflicting or incomplete evidence;
• which safety and formatting validations run after generation.

Citation IDs should originate in the evidence packet, not be invented from free-form model text. Resolve them to a safe canonical URL or document view after validating that the user may still access the source.

RAG improves access to explicit evidence; it does not make the evidence correct or guarantee the model will use it faithfully. AWS's production-level overview similarly describes RAG as an ingestion/index path plus repeated retrieval and generation steps, while leaving product-specific architecture choices to the implementation. See AWS Prescriptive Guidance on RAG components.

Give every boundary a contract

Contracts keep ownership close to the repair. “RAG quality dropped” is not an actionable alert; “candidate recall for error-code queries fell after index version 42” is.

Evaluate retrieval and generation separately

One end-to-end score hides the component to fix. Keep at least these relationships separate:

• retrieved candidates versus the question;
• retrieved candidates versus required evidence;
• final context versus required evidence;
• answer versus supplied evidence;
• answer versus expected outcome;
• citations versus the claims they support.

Build the dataset from real questions, reviewed edge cases, production failures, and deliberately difficult source relationships. Document-derived synthetic questions often reuse source vocabulary and overestimate retrieval performance.

The RAG evaluation metrics guide defines candidate recall, ranking metrics, context coverage, groundedness, correctness, and release gates. AWS also recommends evaluating RAG components rather than treating the generated response as the only signal; see its component evaluation guidance.

Version, trace, and roll back the whole path

For each request, record:

source snapshot / index version
parser, chunker, embedding, and metadata versions
identity and policy version
query-rewrite and retriever versions
candidate set and reranker version
context assembler and prompt version
model and output-validator version
evidence IDs, answer, citations, feedback, latency, and cost

Rollbacks must account for compatibility. Restoring an old index may require the matching embedding and metadata schema. A prompt rollback cannot repair missing evidence. A stale cache can outlive both.

When a user reports a wrong answer, follow the RAG debugging decision tree: prove the source exists and is permitted, then prove it was parsed, indexed, retrieved, assembled, and used correctly—in that order.

Ship the smallest architecture that passes the gates

Start with one authoritative source family, one permission model, and a reviewed evaluation set. A reasonable sequence is:

1. source registry, structured parsing, provenance, and deletion;
2. simple permission-filtered retrieval with measurable recall;
3. bounded context assembly, abstention, and citation mapping;
4. separate retrieval and answer evaluation in CI;
5. task traces, freshness alerts, and rollback;
6. hybrid retrieval, reranking, query rewriting, or multiple indexes only when measured failures justify them.

The correct production architecture is not the diagram with the most boxes. It is the smallest system that preserves evidence and authorization, localizes failures, passes release thresholds, and can be operated by the team that owns it.

Use the stage-contract table to review one planned RAG service. If several boundaries have no owner, version, or test, fix those before selecting another retrieval framework. The JoinAI MasterClass covers the engineering practice behind production retrieval, evaluation, and operation.

Sources and further reading

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• AWS: Understanding Retrieval Augmented Generation
• AWS: Retrieval Augmented Generation options and architectures
• AWS: Evaluating generative AI solutions with multiple retrievers
• AWS: Secure access to data and systems for generative AI