A reranker cannot recover a document the retriever never returned. It can only reorder the candidate set.
That single fact explains most failed reranking projects. Teams retrieve five weak candidates, rerank them, and expect missing evidence to appear. Start with first-stage recall. Add reranking when the evidence is present but ranked too low or surrounded by noise.
Retrieval and reranking solve different problems
The first stage must search a large corpus cheaply. It usually uses BM25, vector similarity, or hybrid retrieval.
The second stage spends more computation on a much smaller set. A cross-encoder or language-model reranker reads the query and each candidate together, which lets it evaluate relationships that independent embeddings compress away.
1,000,000 chunks↓ fast retrieval50 candidates↓ expensive reranking6 context chunks
The architecture trades added latency for better ordering and cleaner context.
Check the prerequisite: candidate recall
For each evaluation query, label the required evidence. Then measure:
candidate_recall@N = queries with required evidence in top N / all queries
If recall@50 is poor, fix indexing, filters, chunking, or query formulation first. If recall@50 is strong and recall@5 is weak, a reranker has room to help.
The RAG evaluation framework separates this retrieval diagnosis from answer groundedness and correctness.
Tune two numbers together
You choose:
- candidate depth N: how many first-stage results enter the reranker;
- context depth K: how many reranked results reach the model.
A practical sweep looks like this:
| Variant | N | K | Measure |
|---|---|---|---|
| Baseline | 8 | 8 | no reranker |
| Small | 20 | 5 | low added latency |
| Medium | 50 | 6 | broader recall |
| Large | 100 | 8 | likely cost ceiling |
Track recall@K, MRR or NDCG, end-to-end answer correctness, P95 latency, and reranking cost. The best ranking metric does not automatically produce the best answer: removing complementary evidence can hurt multi-document questions.
Preserve diversity and document relationships
A reranker may place five near-identical chunks from one document at the top. Add a diversification or grouping rule after scoring:
- maximum 2 chunks per document- merge adjacent chunks from the same section- reserve slots for each required subquery- retain effective-date and permission filters
For multi-part questions, rerank per subquery and then fuse the lists. A single relevance score can favor the easiest part and omit the rest.
Cache carefully
Reranking is often deterministic enough to cache by:
hash(query + candidate_ids + candidate_versions + reranker_version)
Exclude any of those fields and stale results become possible. Permission changes should invalidate or bypass cached rankings. Never reuse a candidate list across tenants merely because the text query matches.
Use a release rule, not a demo
An example gate:
- candidate recall@50 may not regress- recall@5 must improve by at least 4 points overall- no critical category may regress by more than 1 point- P95 added latency must remain below 180 ms- cost per successful answer must stay within budget
Set the actual thresholds from your application. The point is to define the trade before looking at one attractive example.
When to skip reranking
Skip it when the corpus is small enough for precise first-stage search, top results are already strong, latency is strict, or queries are exact lookups. Also skip it while required evidence is absent from the candidate set.
Sometimes better chunking or metadata filters deliver the same lift with less runtime cost.
Reranking checklist
- Required evidence is labeled in the evaluation set.
- First-stage recall is high at the candidate depth.
- Candidate depth and final context depth are tuned separately.
- Ranking metrics and end-to-end answer metrics are both reported.
- Duplicate chunks and document diversity are controlled.
- Cache keys include query, candidates, versions, and authorization scope.
- Added P95 latency and cost have explicit budgets.
