Good RAG interview questions test whether you can design and debug a retrieval system under constraints. They should go beyond defining embeddings or naming a vector database. A strong answer separates retrieval from generation, states what evidence would choose between alternatives, and treats identity, freshness, evaluation, latency, and failure as first-class design inputs.
Use these questions as timed practice. They are not a claim about any company's interview loop. Confirm the actual format with the recruiter.
A scoring rubric for every answer
Score each response from 0 to 2 on six dimensions:
| Dimension | 0 | 1 | 2 |
|---|---|---|---|
| Task contract | Starts with technology | Names a user task | Defines task, refusal, and failure cost |
| Mechanism | Uses labels only | Explains basic flow | Explains contracts and where failure enters |
| Measurement | Says “test it” | Names a metric | Defines dataset, slice, metric, and threshold |
| Tradeoffs | Gives one solution | Names an alternative | Selects using constraints and evidence |
| Security/lifecycle | Omits boundary | Mentions privacy | Locates auth, provenance, retention, and deletion |
| Operations | Omits production | Mentions monitoring | Defines trace, fallback, objective, and rollback |
A score of 9–12 is a strong answer if the claims remain correct under follow-up questions. Do not reward vocabulary without mechanism.
Foundations
1. What problem does RAG solve?
A complete answer says RAG retrieves information from an external source at request time and provides selected context to a generator. It can make answers reflect private, current, or domain-specific material without putting all content in model parameters. It does not guarantee correctness: retrieval can miss, rank poorly, violate access rules, or provide misleading context, and generation can ignore or distort evidence.
Follow-up: when would search results without generation be safer or more useful?
2. Walk through a production RAG request
Cover query validation, identity and tenant scope, query transformation, retrieval, filtering, reranking, context assembly, generation, citation validation, response policy, and tracing. State which components are deterministic and which are probabilistic.
Follow-up: which version identifiers must be stored to reproduce the answer?
3. How do embeddings and lexical search differ?
Explain that lexical methods match terms and can be strong for exact identifiers, rare names, and precise phrases; embedding search represents semantic similarity and can match paraphrases. Hybrid retrieval combines signals, but must be evaluated rather than assumed superior.
Follow-up: design a test slice where lexical search should win.
4. How do you choose chunk boundaries?
Start from document structure and answer granularity. Compare section-aware, sentence, fixed-window, parent-child, and layout-aware approaches. Discuss overlap, metadata, table handling, token cost, and the risk of separating a claim from its qualifier.
Follow-up: which metrics distinguish a chunking failure from a ranking failure?
Retrieval and ranking
5. What does top_k control, and why is more not always better?
More candidates can improve recall but add latency, context cost, and distractors. Distinguish retrieval depth from the smaller number passed after reranking. Choose values with recall curves and end-to-end results on representative queries.
6. When does reranking help?
Reranking helps when a fast first-stage retriever has adequate candidate recall but poor ordering for the task. It cannot recover documents the first stage never retrieved. Measure candidate recall before blaming the reranker.
7. How would you handle filters and access control?
Authorization should constrain retrieval before content reaches the model. Describe server-derived tenant/user identity, allowed-document filters, deny-by-default behavior, cache scoping, and audit evidence. Prompt instructions are not access control.
8. How do you keep an index fresh?
Describe source events or polling, versioned ingestion, idempotent upserts, tombstones or deletion propagation, partial-failure repair, index cutover, and freshness objectives. Include tests for updates and revocation, not only additions.
The RAG freshness and deletion guide provides a deeper failure checklist.
Evaluation and debugging
9. Which retrieval metrics would you use?
For queries with relevance judgments, discuss recall at k, precision at k, reciprocal rank, or nDCG according to whether one or several results matter and whether graded relevance exists. Report slices and uncertainty rather than one aggregate alone.
10. How do you evaluate generated answers?
Separate groundedness or faithfulness, answer relevance, completeness, citation correctness, refusal behavior, and task-specific correctness. Use deterministic checks where possible, calibrated human review where judgment matters, and model judges only with validation against reviewed examples.
Microsoft's current RAG design guide explicitly separates retrieval evaluation from end-to-end language-model evaluation and calls for representative queries, documented hyperparameters, and aggregated results. The RAG evaluation metrics guide provides formulas and dataset fields.
11. The answer is wrong. How do you locate the failure?
Replay the request with stored versions and inspect in order:
- Was the expected source present, current, and authorized?
- Did preprocessing preserve the relevant content and metadata?
- Did the query and filters express the task?
- Did first-stage retrieval include the expected item?
- Did reranking keep it?
- Did context assembly truncate or conflict?
- Did generation use the evidence?
- Did validation or response policy detect the problem?
Add the repaired case to the frozen regression set. The RAG failure-mode decision tree expands this workflow.
12. How do you evaluate questions with no answer in the corpus?
Include unanswerable and adversarial queries as explicit slices. Define acceptable refusal, evidence threshold, and escalation. Measure unsupported-answer rate and false refusal separately; optimizing only answer rate rewards guessing.
Architecture and operations
13. Standard RAG or agentic RAG?
A fixed pipeline fits a predictable query-to-index flow. Agentic retrieval may help when a task needs dynamic source selection, decomposition, multiple searches, or a mix of retrieval and actions. It adds nondeterministic paths, latency, cost, and authorization surface. Microsoft describes the central agentic-RAG decision as exposing retrieval through clearly bounded tools; a strong answer also sets iteration and budget limits.
14. How would you meet a latency objective?
Build a stage budget for authentication, query processing, retrieval, reranking, model time-to-first-token, generation, and validation. Measure tail latency. Consider parallel retrieval, smaller candidate sets, caching with correct identity/version keys, streaming, model routing, and a degraded search-only path.
15. What should a RAG trace contain?
Include request and tenant-safe identifiers, source/index versions, query-transform version, filters, retrieved document IDs and scores, reranker version, selected context IDs, model and prompt version, citations, token/latency/cost fields, validation result, and outcome. Do not log sensitive raw content by default.
16. How would you migrate embedding models?
Treat embeddings and indexes as versioned, incompatible artifacts. Build a shadow index, replay a frozen dataset, compare retrieval and end-to-end slices, dual-read or canary if appropriate, cut over with a rollback pointer, and retire the old version only after the observation window.
Practical exercises
Exercise A: Debugging drill (30 minutes)
You receive 20 failed support queries, request traces, retrieved document IDs, and expected sources. Classify each failure as source, ingestion, query/filter, retrieval, ranking, context, generation, or policy. Propose one repair and one regression test for the largest category.
Deliverable: a failure table and prioritized experiment, not a framework rewrite.
Exercise B: Retrieval evaluation (45 minutes)
Given query relevance judgments and two ranked-result files, compute recall@5, reciprocal rank, and one slice comparison. Choose a retriever and explain what additional evidence could reverse the decision.
Deliverable: tested code plus a short decision note.
Exercise C: Multi-tenant design (45 minutes)
Design document ingestion and answering for 200 organizations. Documents can be updated, shared within an organization, or deleted immediately. Include identity propagation, index/filter strategy, cache keys, deletion verification, and audit fields.
Deliverable: trust-boundary diagram, critical contracts, and three abuse/failure tests.
Exercise D: System-design change (45 minutes)
Start with a fixed support RAG pipeline. Halfway through, add two constraints: some questions require a live order-status API, and p95 latency must remain under the stated product budget. Decide whether to add a bounded tool, an agentic loop, or a separate route.
Deliverable: before/after architecture and a measurable release plan.
How to practice
Answer one question aloud in five minutes, then spend five minutes on follow-ups. Record the answer. Mark every untested assumption and every noun you used without explaining its mechanism. For exercises, keep the code, evaluation result, and design note as portfolio evidence.
Use the AI engineer interview preparation plan to schedule these drills and the JoinAI problem catalog for additional timed practice.
