mlsanityData to 7 Oct 2026

Understanding BERT Rankers Under Distillation

arXiv 2007.11088 ↗First posted 21 Jul 2020cs.IR cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Deep language models such as BERT pre-trained on large corpus have given a huge performance boost to the state-of-the-art information retrieval ranking systems. Knowledge embedded in such models allows them to pick up complex matching signals between passages and queries. However, the high computation cost during inference limits their deployment in real-world search scenarios. In this paper, we study if and how the knowledge for search within BERT can be transferred to a smaller ranker through distillation. Our experiments demonstrate that it is crucial to use a proper distillation procedure, which produces up to nine times speedup while preserving the state-of-the-art performance.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 1; 2023: 1; 2024: 5; 2025: 2; 2026: 2.

15 papers.

Sort:

  1. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  2. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  3. 2024Ranked List Truncation for Large Language Model-based Re-Rankingcited by 20
  4. 2021Teacher Model Fingerprinting Attacks Against Transfer Learningcited by 15
  5. 2021Boosted Dense Retrievercited by 10
  6. 2022Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-rankingcited by 10
  7. 2024Improving Pinterest Search Relevance Using Large Language Modelscited by 8
  8. 2024Self-Calibrated Listwise Reranking with Large Language Modelscited by 7
  9. 2024Best Practices for Distilling Large Language Models into BERT for Web Search Rankingcited by 3
  10. 2024Enhancing Logits Distillation with Plug&Play Kendall's τ Ranking Losscited by 2
  11. 2025MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interactioncited by 1
  12. 2026SERM: Self-Evolving Relevance Model with Agent-Driven Learning from Massive Query Streamscited by 1
3 more
  1. 2026Semantic Search At LinkedIncited by 1
  2. 2023On Elastic Language Modelscited by 0
  3. 2025PLD: A Choice-Theoretic List-Wise Knowledge Distillationcited by 0