Hybrid search and reranking, measured

Ask how to build retrieval for a RAG system and you will usually hear the same recipe: combine keyword search (BM25) with vector search, fuse the results, then rerank the top candidates with a cross-encoder. Every stage sounds like an improvement. Each one also adds latency, cost and moving parts.

I measured the recipe stage by stage on three public test sets from the BEIR benchmark: scientific claims (SciFact), medical questions in plain language (NFCorpus) and financial questions from forums (FiQA). Full method, code and limits are on the LAB/001 page. This article is about what the numbers mean for building real systems.

The short version

1. A strong embedding model on its own was the best system on two of the three datasets, and the best single-stage system on all three.
2. Reciprocal rank fusion with BM25 never improved ranking quality over that model.
3. Reranking helped on one dataset and hurt on two, including with the largest reranker tested, at more than 100 times the latency of the retrieval itself.

None of this means hybrid search or reranking is useless. It means they are hypotheses to test on your data, not defaults to ship.

Why hybrid fusion didn’t help here

Hybrid search exists because lexical and dense retrieval fail differently. BM25 finds exact identifiers, names and rare terms; embeddings find paraphrases. Fusing them should cover both blind spots.

That only works when both retrievers are reasonably good. Reciprocal rank fusion gives each list equal weight. When one retriever is much weaker (here BM25 trails the embedding model by 0.15 nDCG@10 on FiQA), fusion pulls weak results into the top ranks about as often as it rescues good ones. The per-query count shows it directly:

On SciFact, fusion improved 53 queries and worsened 53. On FiQA it worsened more queries (216) than it improved (156).

Where hybrid still makes sense:

• Queries full of identifiers: part numbers, error codes, legal references, product SKUs. These are the cases embeddings handle worst, and they are underrepresented in these three datasets.
• As a recall stage before a good reranker: hybrid raised Recall@100 on SciFact, so a reranker had more relevant candidates to work with.
• With weighting: a weighted fusion, or a lexical stage that only contributes when the query contains rare tokens, avoids diluting a strong dense ranking. I did not test this here; it is the obvious next experiment.

Why reranking was a coin flip

Cross-encoders read the query and each candidate together, which in principle makes them more accurate than retrieval models that encode them separately. In practice they are trained on particular kinds of data, mostly web search questions and answers, and their advantage does not always transfer.

• The two smaller rerankers (22 and 278 million parameters) lowered nDCG@10 in five of six cases.
• The largest, bge-reranker-v2-m3 (568 million parameters), lifted FiQA from 0.406 to 0.436 nDCG@10, about 7% relative. Financial forum questions are close to the question-answer style rerankers are trained on.
• The same model lowered SciFact from 0.744 to 0.733 and NFCorpus from 0.375 to 0.341. Scientific claims and medical topic queries are a different task from “find the passage that answers this question”.

And the cost: reranking 100 candidates took a median of 540–670 ms per query on an RTX 4070, against roughly 4 ms for the dense search. Fewer candidates or a smaller model cut that, at some cost in quality. Either way it is a real trade-off that belongs in the decision.

What I would do on a real project

1. Build an evaluation set first. Fifty to a hundred real questions with the passages that answer them (see how to evaluate a RAG system). Without it, none of these choices can be made on evidence.
2. Start with a strong embedding model and measure it.
3. Look at the failures. If they involve exact identifiers or rare terms, try adding lexical search, ideally weighted, and check the per-query counts, not just the average.
4. Try a reranker only if ranking is the problem: high Recall@100 but low nDCG@10. Test it on your queries, and decide whether the gain is worth the latency.
5. Re-measure when anything changes: documents, models or the kind of questions users ask.

The common advice is a sensible list of things to try. Measured on your own data, some of them will pay for themselves and some won’t, and you can only find out which by testing.