# ABNIRML: Analyzing the Behavior of Neural IR Models

arXiv: https://arxiv.org/abs/2011.00696 · HTML: https://mlsanity.com/paper/2011.00696/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2020.

## Built on by (most cited first, 23 of 23)

- ["According to ...": Prompting Language Models Improves Quoting from Pre-Training Data](https://mlsanity.com/paper/2305.13252/) (2023)
- [Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2502.06872/) (2025)
- [NevIR: Negation in Neural Information Retrieval](https://mlsanity.com/paper/2305.07614/) (2023)
- [What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary](https://mlsanity.com/paper/2212.10380/) (2022)
- [When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets](https://mlsanity.com/paper/2309.08541/) (2023)
- [Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval](https://mlsanity.com/paper/2212.10526/) (2022)
- [Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence](https://mlsanity.com/paper/2503.05037/) (2025)
- [Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2407.06992/) (2024)
- [DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queries](https://mlsanity.com/paper/2310.04678/) (2023)
- [Match Your Words! A Study of Lexical Matching in Neural Information Retrieval](https://mlsanity.com/paper/2112.05662/) (2021)
- [An Attribution Method for Siamese Encoders](https://mlsanity.com/paper/2310.05703/) (2023)
- [Representation biases in sentence transformers](https://mlsanity.com/paper/2301.13039/) (2023)
- [Approximate Attributions for Off-the-Shelf Siamese Transformers](https://mlsanity.com/paper/2402.02883/) (2024)
- [How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective](https://mlsanity.com/paper/2504.07898/) (2025)
- [Pathway to Relevance: How Cross-Encoders Implement a Semantic Variant of BM25](https://mlsanity.com/paper/2502.04645/) (2025)
- [Entity-Conditioned Question Generation for Robust Attention Distribution in Neural Information Retrieval](https://mlsanity.com/paper/2204.11373/) (2022)
- [Positional Bias in Long-Document Ranking: Impact, Assessment, and Mitigation](https://mlsanity.com/paper/2207.01262/) (2022)
- [Probing Ranking LLMs: A Mechanistic Analysis for Information Retrieval](https://mlsanity.com/paper/2410.18527/) (2024)
- [The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through](https://mlsanity.com/paper/2203.15364/) (2022)
- [The Role of Complex NLP in Transformers for Text Ranking?](https://mlsanity.com/paper/2207.02522/) (2022)
- [Explainability of Text Processing and Retrieval Methods: A Survey](https://mlsanity.com/paper/2212.07126/) (2022)
- [Matching Meaning at Scale: Evaluating Semantic Search for 18th-Century Intellectual History through the Case of Locke](https://mlsanity.com/paper/2605.09236/) (2026)
- [Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors](https://mlsanity.com/paper/2606.02814/) (2026)

## Abstract (arXiv)

Pretrained contextualized language models such as BERT and T5 have established a new state-of-the-art for ad-hoc search. However, it is not yet well-understood why these methods are so effective, what makes some variants more effective than others, and what pitfalls they may have. We present a new comprehensive framework for Analyzing the Behavior of Neural IR ModeLs (ABNIRML), which includes new types of diagnostic probes that allow us to test several characteristics -- such as writing styles, factuality, sensitivity to paraphrasing and word order -- that are not addressed by previous techniques. To demonstrate the value of the framework, we conduct an extensive empirical study that yields insights into the factors that contribute to the neural model's gains, and identify potential unintended biases the models exhibit. Some of our results confirm conventional wisdom, like that recent neural ranking models rely less on exact term overlap with the query, and instead leverage richer linguistic information, evidenced by their higher sensitivity to word and sentence order. Other results are more surprising, such as that some models (e.g., T5 and ColBERT) are biased towards factually correct (rather than simply relevant) texts. Further, some characteristics vary even for the same base language model, and other characteristics can appear due to random variations during model training.