# Understanding BERT Rankers Under Distillation

arXiv: https://arxiv.org/abs/2007.11088 · HTML: https://mlsanity.com/paper/2007.11088/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jul 2020.

## Built on by (most cited first, 15 of 15)

- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset](https://mlsanity.com/paper/2108.13897/) (2021)
- [Ranked List Truncation for Large Language Model-based Re-Ranking](https://mlsanity.com/paper/2404.18185/) (2024)
- [Teacher Model Fingerprinting Attacks Against Transfer Learning](https://mlsanity.com/paper/2106.12478/) (2021)
- [Boosted Dense Retriever](https://mlsanity.com/paper/2112.07771/) (2021)
- [Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-ranking](https://mlsanity.com/paper/2204.11673/) (2022)
- [Improving Pinterest Search Relevance Using Large Language Models](https://mlsanity.com/paper/2410.17152/) (2024)
- [Self-Calibrated Listwise Reranking with Large Language Models](https://mlsanity.com/paper/2411.04602/) (2024)
- [Enhancing Logits Distillation with Plug&Play Kendall's τ Ranking Loss](https://mlsanity.com/paper/2409.17823/) (2024)
- [Best Practices for Distilling Large Language Models into BERT for Web Search Ranking](https://mlsanity.com/paper/2411.04539/) (2024)
- [MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interaction](https://mlsanity.com/paper/2512.07846/) (2025)
- [SERM: Self-Evolving Relevance Model with Agent-Driven Learning from Massive Query Streams](https://mlsanity.com/paper/2601.09515/) (2026)
- [Semantic Search At LinkedIn](https://mlsanity.com/paper/2602.07309/) (2026)
- [On Elastic Language Models](https://mlsanity.com/paper/2311.07204/) (2023)
- [PLD: A Choice-Theoretic List-Wise Knowledge Distillation](https://mlsanity.com/paper/2506.12542/) (2025)

## Abstract (arXiv)

Deep language models such as BERT pre-trained on large corpus have given a huge performance boost to the state-of-the-art information retrieval ranking systems. Knowledge embedded in such models allows them to pick up complex matching signals between passages and queries. However, the high computation cost during inference limits their deployment in real-world search scenarios. In this paper, we study if and how the knowledge for search within BERT can be transferred to a smaller ranker through distillation. Our experiments demonstrate that it is crucial to use a proper distillation procedure, which produces up to nine times speedup while preserving the state-of-the-art performance.