# Match Your Words! A Study of Lexical Matching in Neural Information Retrieval

arXiv: https://arxiv.org/abs/2112.05662 · HTML: https://mlsanity.com/paper/2112.05662/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 10 Dec 2021.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking](https://mlsanity.com/paper/2107.05720/) (2021)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling](https://mlsanity.com/paper/2104.06967/) (2021)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [Simple Entity-Centric Questions Challenge Dense Retrievers](https://mlsanity.com/paper/2109.08535/) (2021)
- [SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval](https://mlsanity.com/paper/2109.10086/) (2021)
- [ABNIRML: Analyzing the Behavior of Neural IR Models](https://mlsanity.com/paper/2011.00696/) (2020)
- [A White Box Analysis of ColBERT](https://mlsanity.com/paper/2012.09650/) (2020)

## Built on by (most cited first, 10 of 10)

- [From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective](https://mlsanity.com/paper/2205.04733/) (2022)
- [When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets](https://mlsanity.com/paper/2309.08541/) (2023)
- [How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective](https://mlsanity.com/paper/2504.07898/) (2025)
- [Pathway to Relevance: How Cross-Encoders Implement a Semantic Variant of BM25](https://mlsanity.com/paper/2502.04645/) (2025)
- [Learning Retrieval Models with Sparse Autoencoders](https://mlsanity.com/paper/2603.13277/) (2026)
- [Unsupervised Domain Adaptation for Sparse Retrieval by Filling Vocabulary and Word Frequency Gaps](https://mlsanity.com/paper/2211.03988/) (2022)
- [Probing Ranking LLMs: A Mechanistic Analysis for Information Retrieval](https://mlsanity.com/paper/2410.18527/) (2024)
- [Can a Frozen Pretrained Language Model be used for Zero-shot Neural Retrieval on Entity-centric Questions?](https://mlsanity.com/paper/2303.05153/) (2023)
- [Blending Learning to Rank and Dense Representations for Efficient and Effective Cascades](https://mlsanity.com/paper/2510.16393/) (2025)
- [Intelligent Scientific Literature Explorer using Machine Learning (ISLE)](https://mlsanity.com/paper/2512.12760/) (2025)

## Abstract (arXiv)

Neural Information Retrieval models hold the promise to replace lexical matching models, e.g. BM25, in modern search engines. While their capabilities have fully shone on in-domain datasets like MS MARCO, they have recently been challenged on out-of-domain zero-shot settings (BEIR benchmark), questioning their actual generalization capabilities compared to bag-of-words approaches. Particularly, we wonder if these shortcomings could (partly) be the consequence of the inability of neural IR models to perform lexical matching off-the-shelf. In this work, we propose a measure of discrepancy between the lexical matching performed by any (neural) model and an 'ideal' one. Based on this, we study the behavior of different state-of-the-art neural IR models, focusing on whether they are able to perform lexical matching when it's actually useful, i.e. for important terms. Overall, we show that neural IR models fail to properly generalize term importance on out-of-domain collections or terms almost unseen during training