# Understanding the Behaviors of BERT in Ranking

arXiv: https://arxiv.org/abs/1904.07531 · HTML: https://mlsanity.com/paper/1904.07531/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2019.

## Built on by (most cited first, 30 of 49)

- [RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking](https://mlsanity.com/paper/2110.07367/) (2021)
- [Wider and Deeper LLM Networks are Fairer LLM Evaluators](https://mlsanity.com/paper/2308.01862/) (2023)
- [HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrieval](https://mlsanity.com/paper/2103.15049/) (2021)
- [Curriculum Learning for Dense Retrieval Distillation](https://mlsanity.com/paper/2204.13679/) (2022)
- [R2-D2: A Modular Baseline for Open-Domain Question Answering](https://mlsanity.com/paper/2109.03502/) (2021)
- [Towards Robust Ranker for Text Retrieval](https://mlsanity.com/paper/2206.08063/) (2022)
- [A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment](https://mlsanity.com/paper/2308.05696/) (2023)
- [A Semantic Search Engine for Mathlib4](https://mlsanity.com/paper/2403.13310/) (2024)
- [Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2502.06872/) (2025)
- [Rider: Reader-Guided Passage Reranking for Open-Domain Question Answering](https://mlsanity.com/paper/2101.00294/) (2021)
- [Learning to Relate to Previous Turns in Conversational Search](https://mlsanity.com/paper/2306.02553/) (2023)
- [Emotion Based Hate Speech Detection using Multimodal Learning](https://mlsanity.com/paper/2202.06218/) (2022)
- [Neural ranking models for document retrieval](https://mlsanity.com/paper/2102.11903/) (2021)
- [Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAG](https://mlsanity.com/paper/2409.07691/) (2024)
- [Natural Language Processing for Smart Healthcare](https://mlsanity.com/paper/2110.15803/) (2021)
- [GripRank: Bridging the Gap between Retrieval and Generation via the Generative Knowledge Improved Passage Ranking](https://mlsanity.com/paper/2305.18144/) (2023)
- [Improving Pretrained Models for Zero-shot Multi-label Text Classification through Reinforced Label Hierarchy Reasoning](https://mlsanity.com/paper/2104.01666/) (2021)
- [Mixed Attention Transformer for Leveraging Word-Level Knowledge to Neural Cross-Lingual Information Retrieval](https://mlsanity.com/paper/2109.02789/) (2021)
- [P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning](https://mlsanity.com/paper/2205.01886/) (2022)
- [Pruning the Index Contents for Memory Efficient Open-Domain QA](https://mlsanity.com/paper/2102.10697/) (2021)
- [Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding](https://mlsanity.com/paper/2204.02922/) (2022)
- [Generating Natural Language Queries for More Effective Systematic Review Screening Prioritisation](https://mlsanity.com/paper/2309.05238/) (2023)
- [Pathway to Relevance: How Cross-Encoders Implement a Semantic Variant of BM25](https://mlsanity.com/paper/2502.04645/) (2025)
- [Training Adaptive Computation for Open-Domain Question Answering with Computational Constraints](https://mlsanity.com/paper/2107.02102/) (2021)
- [Hard Cases Detection in Motion Prediction by Vision-Language Foundation Models](https://mlsanity.com/paper/2405.20991/) (2024)
- [Deep Bag-of-Words Model: An Efficient and Interpretable Relevance Architecture for Chinese E-Commerce](https://mlsanity.com/paper/2407.09395/) (2024)
- [Learning an Effective Premise Retrieval Model for Efficient Mathematical Formalization](https://mlsanity.com/paper/2501.13959/) (2025)
- [News Article Retrieval in Context for Event-centric Narrative Creation](https://mlsanity.com/paper/2106.16053/) (2021)
- [Modeling Relevance Ranking under the Pre-training and Fine-tuning Paradigm](https://mlsanity.com/paper/2108.05652/) (2021)
- [Block-SCL: Blocking Matters for Supervised Contrastive Learning in Product Matching](https://mlsanity.com/paper/2207.02008/) (2022)

## Abstract (arXiv)

This paper studies the performances and behaviors of BERT in ranking tasks. We explore several different ways to leverage the pre-trained BERT and fine-tune it on two ranking tasks: MS MARCO passage reranking and TREC Web Track ad hoc document ranking. Experimental results on MS MARCO demonstrate the strong effectiveness of BERT in question-answering focused passage ranking tasks, as well as the fact that BERT is a strong interaction-based seq2seq matching model. Experimental results on TREC show the gaps between the BERT pre-trained on surrounding contexts and the needs of ad hoc document ranking. Analyses illustrate how BERT allocates its attentions between query-document tokens in its Transformer layers, how it prefers semantic matches between paraphrase tokens, and how that differs with the soft match patterns learned by a click-trained neural ranker.