Reranking
A second, slower pass that reorders retrieved results by actually reading them — usually the cheapest large improvement available to a RAG system.
When not to use it
- When first-stage recall is the problem. If the right passage isn't in the candidate set, reranking cannot help. Measure recall@k first; if it's low, fix retrieval or chunking instead.
- When latency is genuinely tight. Sub-100ms budgets may not have room. Be honest about whether yours actually is.
- When you retrieve three passages and use three. There's nothing to rerank. Reranking needs a surplus to discard.
- When chunking is broken. Same reasoning as recall: a reranker cannot reorder its way to an answer that no chunk contains.
Reach for something else instead
- Hybrid retrieval — BM25 plus dense. Improves first-stage recall, which reranking cannot.
- Better chunking — often the actual problem, and free.
- LLM-as-reranker — hand the passages to a model and ask it to order them. Works; costs more.
- Late interaction (ColBERT-style) — one stage, between the two in cost and accuracy.
Sources & further reading
- Nogueira & Cho (2019), Passage Re-ranking with BERT — the paper that made cross-encoder reranking standard.
- Khattab & Zaharia (2020), ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT — the middle ground between bi- and cross-encoders.
- Robertson & Zaragoza (2009), The Probabilistic Relevance Framework: BM25 and Beyond — the first-stage retriever a reranker most often sits on top of.
Primary sources, listed so you can check the claims on this page rather than take them on trust.
Where people go wrong
- Reranking a candidate set that's too small. Retrieve 50, rerank to 5 — not retrieve 5, rerank to 5.
- Adding a reranker while first-stage recall is unmeasured, then not knowing whether it helped or why.
- Assuming it fixes hallucination. If retrieval never found the answer, better ordering of wrong passages produces a better-ordered wrong answer.
- Passing all 50 reranked passages to the model. Crowding is real; the point of reranking is to discard.
At a glance
FieldLanguage & LLMs
Stagesecond pass, after retrieval
Model typecross-encoder
Typical shaperetrieve 50 → rerank → keep 3–5
Latency cost~50–200ms
Ceilingfirst-stage recall@k
DifficultyIntermediate
Flashcards for this concept
Question
—
Answer
—
1 / 4
Often compared with
Reranking vs. better embeddings — reranking improves the order of what you found; better embeddings improve what you find. If the answer isn't in the candidates, only the second one helps.