Retrieval Observatory

Most advice about retrieval-augmented generation is folklore: always use hybrid search, always add a reranker. Those are reasonable defaults, but they are claims about data, and they can be measured. This Lab project measures them.

What is measured

Each query goes through the stages a typical RAG retriever is built from, and quality is measured at the output of every stage:

• BM25: classic lexical search with stemming.
• Dense: two bi-encoder embedding models (bge-small and bge-base), exact cosine search.
• Hybrid: reciprocal rank fusion of BM25 and the bge-base dense results.
• Reranked: cross-encoders of three sizes reordering the hybrid top 100. The largest also reranks the dense top 100, to separate the effect of fusion from the effect of reranking.

The data is three public test sets from the BEIR benchmark, chosen because they differ: scientific claims (SciFact), medical questions in plain language (NFCorpus) and financial opinion questions (FiQA).

Results

What the numbers say

1. A good embedding model alone was the strongest single choice. Dense retrieval with bge-base beat BM25 on all three datasets, by a wide margin on FiQA (0.406 against 0.251 nDCG@10), and had the best nDCG@10 of any system on SciFact and NFCorpus.

2. Hybrid fusion did not beat dense retrieval on ranking quality anywhere. Per query it mostly reshuffled results: on SciFact it improved 53 queries and worsened 53; on FiQA it improved 156 and worsened 216. With equal-weight fusion, a much weaker retriever pulls good dense results down. Hybrid did raise Recall@100 on SciFact (0.970 against 0.963), which matters if a reranker follows, but not on the other two.

3. Rerankers depend on the domain. The two smaller cross-encoders lowered nDCG@10 in five of six cases. The largest, bge-reranker-v2-m3, lifted FiQA from 0.406 to 0.436 (about 7% relative) but lowered SciFact (0.744 → 0.733) and NFCorpus (0.375 → 0.341). It also costs roughly 540–670 ms per query to rerank 100 candidates on this GPU, against about 4 ms for the dense search itself.

The practical conclusion: the textbook pipeline of hybrid search plus a reranker is a hypothesis to test, not a default to ship. Start from a strong embedding model, measure on your own questions, and add each stage only where it pays for its latency.

Averages hide a lot. The table below counts, query by query, how often a change actually helped:

Reproducing it

The code is a standalone Python package (labs/retrieval-observatory in this site’s repository). After downloading the public datasets and models, one command runs every stage and writes the JSON file these figures are rendered from. The metric implementation was checked against published figures: BM25 and both embedding models reproduce reported SciFact and NFCorpus scores to two or three decimal places.

Limits

Three English datasets, one machine, exact search rather than an approximate index, and models from 22 to 568 million parameters, and the BEIR relevance judgements, which are known to be incomplete (an unjudged relevant document counts as a miss). Larger rerankers, domain-tuned models or weighted fusion could change the picture. On your own documents, the only way to know is to measure, which is exactly what this harness is for.