# Pathway to Relevance: How Cross-Encoders Implement a Semantic Variant of BM25

arXiv: https://arxiv.org/abs/2502.04645 · HTML: https://mlsanity.com/paper/2502.04645/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 7 Feb 2025.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [Localizing Model Behavior with Path Patching](https://mlsanity.com/paper/2304.05969/) (2023)
- [A Deep Relevance Matching Model for Ad-hoc Retrieval](https://mlsanity.com/paper/1711.08611/) (2017)
- [Text Matching as Image Recognition](https://mlsanity.com/paper/1602.06359/) (2016)
- [Understanding the Behaviors of BERT in Ranking](https://mlsanity.com/paper/1904.07531/) (2019)
- [ABNIRML: Analyzing the Behavior of Neural IR Models](https://mlsanity.com/paper/2011.00696/) (2020)
- [A White Box Analysis of ColBERT](https://mlsanity.com/paper/2012.09650/) (2020)
- [Match Your Words! A Study of Lexical Matching in Neural Information Retrieval](https://mlsanity.com/paper/2112.05662/) (2021)
- [Axiomatic Causal Interventions for Reverse Engineering Relevance Computation in Neural Retrieval Models](https://mlsanity.com/paper/2405.02503/) (2024)
- [Diagnosing BERT with Retrieval Heuristics](https://mlsanity.com/paper/2201.04458/) (2022)
- [Finding Inverse Document Frequency Information in BERT](https://mlsanity.com/paper/2202.12191/) (2022)

## Built on by (most cited first, 6 of 6)

- [Taxonomy of the Retrieval System Framework: Pitfalls and Paradigms](https://mlsanity.com/paper/2601.20131/) (2026)
- [When Fine-Tuning Fails: Lessons from MS MARCO Passage Ranking](https://mlsanity.com/paper/2506.18535/) (2025)
- [Automating AI Failure Tracking: Semantic Association of Reports in AI Incident Database](https://mlsanity.com/paper/2507.23669/) (2025)
- [Retrieval-Augmented Detection of Potentially Abusive Clauses in Chilean Terms of Service](https://mlsanity.com/paper/2605.26019/) (2026)
- [The Emergence of Relevance Through Axiomatic Attention Patterns During LoRA Fine-Tuning](https://mlsanity.com/paper/2608.23338/) (2026)
- [A Mechanistic Study of AI-Text Detection Neurons in Frozen BERT: Sparse Probing and Activation Patching on RAID](https://mlsanity.com/paper/2609.30287/) (2026)

## Abstract (arXiv)

Mechanistic interpretation has greatly contributed to a more detailed understanding of generative language models, enabling significant progress in identifying structures that implement key behaviors through interactions between internal components. In contrast, interpretability in information retrieval (IR) remains relatively coarse-grained, and much is still unknown as to how IR models determine whether a document is relevant to a query. In this work, we address this gap by mechanistically analyzing how one commonly used model, a cross-encoder, estimates relevance. We find that the model extracts traditional relevance signals, such as term frequency and inverse document frequency, in early-to-middle layers. These concepts are then combined in later layers, similar to the well-known probabilistic ranking function, BM25. Overall, our analysis offers a more nuanced understanding of how IR models compute relevance. Isolating these components lays the groundwork for future interventions that could enhance transparency, mitigate safety risks, and improve scalability.