# Per-Query Gating of LLM Rerankers for Multi-Hop Retrieval

arXiv: https://arxiv.org/abs/2609.22880 · HTML: https://mlsanity.com/paper/2609.22880/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 19 Sep 2026.

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity](https://mlsanity.com/paper/2403.14403/) (2024)
- [From RAG to Memory: Non-Parametric Continual Learning for Large Language Models](https://mlsanity.com/paper/2502.14802/) (2025)
- [RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!](https://mlsanity.com/paper/2312.02724/) (2023)
- [Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach](https://mlsanity.com/paper/2407.16833/) (2024)
- [PropRAG: Guiding Retrieval with Beam Search over Proposition Paths](https://mlsanity.com/paper/2504.18070/) (2025)
- [Retrieval as a Decision: Training-Free Adaptive Gating for Efficient RAG](https://mlsanity.com/paper/2511.09803/) (2025)
- [Calibrated Fusion for Heterogeneous Graph-Vector Retrieval in Multi-Hop QA](https://mlsanity.com/paper/2603.28886/) (2026)
- [Lightweight Query Routing for Adaptive RAG: A Baseline Study on RAGRouter-Bench](https://mlsanity.com/paper/2604.03455/) (2026)
- [Cost-Aware Query Routing in RAG: Empirical Analysis of Retrieval Depth Tradeoffs](https://mlsanity.com/paper/2606.02581/) (2026)
- [Regime-Conditional Retrieval: Theory and a Transferable Router for Two-Hop QA](https://mlsanity.com/paper/2604.09019/) (2026)
- [When Reranking Hurts: Uncertainty-Based Gating for Few-Shot Reranking](https://mlsanity.com/paper/2606.31087/) (2026)
- [Predictable Failure in Multi-Hop Retrieval: Score-Distributional Confidence Scoring and Abstention](https://mlsanity.com/paper/2609.22056/) (2026)

## Abstract (arXiv)

LLM rerankers add of the order of 0.2-0.3 per 1,000 queries and about a second of tail latency on top of a graph-augmented dense pipeline such as HippoRAG2, and on three multi-hop benchmarks they improve final-hop top-K coverage on seven of nine (dataset, K) cells, by up to +34.8 pp. We ask whether a learned per-query gate can skip the reranker where it will not help, using only features available before the LLM call (27 score and lexical statistics of the two retrieval lists plus a PCA of a small query embedding) with an executable fallback. Every choice, including the fallback and the threshold, is made inside the training fold and applied once to held-out queries, and harmful skips (the rerank would have found the target, the fallback did not) are reported next to the aggregate coverage. Across nine cells on 2WikiMultiHopQA, MuSiQue and HotpotQA the gate skips 51% of calls at an average held-out LastHop@K cost of 1.2 pp; four cells meet a pre-registered 1 pp rule, harmful skips occur in eight (190 harmful against 136 beneficial), and a random gate at the same skip rate loses 2 to 11 pp on the high-lift cells. A second rule sets each cell's threshold from a pre-specified budget on the expected harmful-skip rate over Platt-calibrated harm probabilities (ECE 0.025 after calibration, 0.094 before): at a 1 pp budget the gate skips 42% at -0.8 pp with 66 harmful skips and six cells within 1 pp, but realised harm exceeds the promise in six cells (mean 1.45 vs 0.83 pp), a selection optimism we quantify; a 0.5 pp budget realises about 1 pp. The harm probabilities are calibrated but barely discriminative (AUC 0.16 to 0.70). An earlier version reported 73% "lossless" savings; that figure rested on an oracle fallback and a wrong MuSiQue target, and we document both.