# Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline

arXiv: https://arxiv.org/abs/2101.08751 · HTML: https://mlsanity.com/paper/2101.08751/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jan 2021.

## Built on by (most cited first, 30 of 65)

- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval](https://mlsanity.com/paper/2407.19669/) (2024)
- [MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrieval](https://mlsanity.com/paper/2307.00589/) (2023)
- [RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking](https://mlsanity.com/paper/2110.07367/) (2021)
- [RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses](https://mlsanity.com/paper/2210.10634/) (2022)
- [COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List](https://mlsanity.com/paper/2104.07186/) (2021)
- [Zero-Shot Listwise Document Reranking with a Large Language Model](https://mlsanity.com/paper/2305.02156/) (2023)
- [RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models](https://mlsanity.com/paper/2309.15088/) (2023)
- [Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation](https://mlsanity.com/paper/2206.10128/) (2022)
- [JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking](https://mlsanity.com/paper/2411.00142/) (2024)
- [Fine-Grained Distillation for Long Document Retrieval](https://mlsanity.com/paper/2212.10423/) (2022)
- [WebBrain: Learning to Generate Factually Correct Articles for Queries by Grounding on Large Web Corpus](https://mlsanity.com/paper/2304.04358/) (2023)
- [Multi-CPR: A Multi Domain Chinese Dataset for Passage Retrieval](https://mlsanity.com/paper/2203.03367/) (2022)
- [In Defense of Cross-Encoders for Zero-Shot Retrieval](https://mlsanity.com/paper/2212.06121/) (2022)
- [HLATR: Enhance Multi-stage Text Retrieval with Hybrid List Aware Transformer Reranking](https://mlsanity.com/paper/2205.10569/) (2022)
- [Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queries](https://mlsanity.com/paper/2505.15420/) (2025)
- [Multi-Head RAG: Solving Multi-Aspect Problems with LLMs](https://mlsanity.com/paper/2406.05085/) (2024)
- [Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translation](https://mlsanity.com/paper/2402.17144/) (2024)
- [DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions](https://mlsanity.com/paper/2305.16636/) (2023)
- [Wikiformer: Pre-training with Structured Information of Wikipedia for Ad-hoc Retrieval](https://mlsanity.com/paper/2312.10661/) (2023)
- [RankMamba: Benchmarking Mamba's Document Ranking Performance in the Era of Transformers](https://mlsanity.com/paper/2403.18276/) (2024)
- [Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models](https://mlsanity.com/paper/2312.02969/) (2023)
- [Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions](https://mlsanity.com/paper/2305.10783/) (2023)
- [MGeo: Multi-Modal Geographic Pre-Training Method](https://mlsanity.com/paper/2301.04283/) (2023)
- [THUIR2 at NTCIR-16 Session Search (SS) Task](https://mlsanity.com/paper/2307.00250/) (2023)
- [Metadata-Induced Contrastive Learning for Zero-Shot Multi-Label Text Classification](https://mlsanity.com/paper/2202.05932/) (2022)
- [STARD: A Chinese Statute Retrieval Dataset with Real Queries Issued by Non-professionals](https://mlsanity.com/paper/2406.15313/) (2024)
- [State Space Models are Strong Text Rerankers](https://mlsanity.com/paper/2412.14354/) (2024)
- [Long Document Re-ranking with Modular Re-ranker](https://mlsanity.com/paper/2205.04275/) (2022)
- [Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search](https://mlsanity.com/paper/2212.09017/) (2022)

## Abstract (arXiv)

Pre-trained deep language models~(LM) have advanced the state-of-the-art of text retrieval. Rerankers fine-tuned from deep LM estimates candidate relevance based on rich contextualized matching signals. Meanwhile, deep LMs can also be leveraged to improve search index, building retrievers with better recall. One would expect a straightforward combination of both in a pipeline to have additive performance gain. In this paper, we discover otherwise and that popular reranker cannot fully exploit the improved retrieval result. We, therefore, propose a Localized Contrastive Estimation (LCE) for training rerankers and demonstrate it significantly improves deep two-stage models.