mlsanityData to 14 Oct 2026

Drowning in Documents: Consequences of Scaling Reranker Inference

arXiv 2411.11767 ↗First posted 18 Nov 2024cs.IR cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Rerankers, typically cross-encoders, are computationally intensive but are frequently used because they are widely assumed to outperform cheaper initial IR systems. We challenge this assumption by measuring reranker performance for full retrieval, not just re-scoring first-stage retrieval. To provide a more robust evaluation, we prioritize strong first-stage retrieval using modern dense embeddings and test rerankers on a variety of carefully chosen, challenging tasks, including internally curated datasets to avoid contamination, and out-of-domain ones. Our empirical results reveal a surprising trend: the best existing rerankers provide initial improvements when scoring progressively more documents, but their effectiveness gradually declines and can even degrade quality beyond a certain limit. We hope that our findings will spur future research to improve reranking.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 51 of its 71 references that are in the map. First posted on arXiv 18 Nov 2024.

Builds on

51 of 71 references.

References · 51

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2017Billion-scale similarity search with GPUscited by 2,497
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  7. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  9. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  10. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  11. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  12. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
38 more
  1. 2019Passage Re-ranking with BERTcited by 645
  2. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  3. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  5. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
  6. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
  7. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 267
  8. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  9. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 214
  10. 2019Multi-Stage Document Ranking with BERTcited by 201
  11. 2019Certified Robustness to Adversarial Word Substitutionscited by 168
  12. 2019Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoringcited by 165
  13. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 158
  14. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 155
  15. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  16. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
  17. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  18. 2021Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipelinecited by 65
  19. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  20. 2018Learning a Deep Listwise Context Model for Ranking Refinementcited by 62
  21. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 48
  22. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  23. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
  24. 2024Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Trackcited by 32
  25. 2024Scaling Laws For Dense Retrievalcited by 29
  26. 2024Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Modelscited by 23
  27. 2018Learning Groupwise Multivariate Scoring Functions Using Deep Neural Networkscited by 22
  28. 2020Interpretable & Time-Budget-Constrained Contextualization for Re-Rankingcited by 21
  29. 2022In Defense of Cross-Encoders for Zero-Shot Retrievalcited by 21
  30. 2022RELIC: Retrieving Evidence for Literary Claimscited by 18
  31. 2024BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectivescited by 15
  32. 2024Top-Down Partitioning for Efficient List-Wise Rankingcited by 13
  33. 2023Can Query Expansion Improve Generalization of Strong Cross-Encoder Rankers?cited by 12
  34. 2020Choppy: Cut Transformer For Ranked List Truncationcited by 11
  35. 2018Strength in Numbers: Trading-off Robustness and Computation via Adversarially-Trained Ensemblescited by 8
  36. 2021A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Modelscited by 8
  37. 2023Policy-Gradient Training of Language Models for Rankingcited by 5
  38. 2024InstUPR : Instruction-based Unsupervised Passage Reranking with Large Language Modelscited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 1.

8 papers.

  1. 2025FinSage: A Multi-aspect RAG System for Financial Filings Question Answeringcited by 19
  2. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  3. 2025ORANSight-2.0: Foundational LLMs for O-RANcited by 13
  4. 2025ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Rerankingcited by 11
  5. 2025AI5GTest: AI-Driven Specification-Aware Automated Testing and Validation of 5G O-RAN Componentscited by 4
  6. 2025QuOTE: Question-Oriented Text Embeddingscited by 2
  7. 2026Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scalecited by 2
  8. 2025JointRank: Rank Large Set with Single Passcited by 1