# A Linguistic Study on Relevance Modeling in Information Retrieval

arXiv: https://arxiv.org/abs/2103.00956 · HTML: https://mlsanity.com/paper/2103.00956/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2021.

## Built on by (most cited first, 6 of 6)

- [B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval](https://mlsanity.com/paper/2104.09791/) (2021)
- [Match-Prompt: Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learning](https://mlsanity.com/paper/2204.02725/) (2022)
- [A Cooperative Multi-Agent Framework for Zero-Shot Named Entity Recognition](https://mlsanity.com/paper/2502.18702/) (2025)
- [Benchmark Granularity and Model Robustness for Image-Text Retrieval](https://mlsanity.com/paper/2407.15239/) (2024)
- [Boosting LLM-based Relevance Modeling with Distribution-Aware Robust Learning](https://mlsanity.com/paper/2412.12504/) (2024)
- [Probing Ranking LLMs: A Mechanistic Analysis for Information Retrieval](https://mlsanity.com/paper/2410.18527/) (2024)

## Abstract (arXiv)

Relevance plays a central role in information retrieval (IR), which has received extensive studies starting from the 20th century. The definition and the modeling of relevance has always been critical challenges in both information science and computer science research areas. Along with the debate and exploration on relevance, IR has already become a core task in many real-world applications, such as Web search engines, question answering systems, conversational bots, and so on. While relevance acts as a unified concept in all these retrieval tasks, the inherent definitions are quite different due to the heterogeneity of these tasks. This raises a question to us: Do these different forms of relevance really lead to different modeling focuses? To answer this question, in this work, we conduct an empirical study on relevance modeling in three representative IR tasks, i.e., document retrieval, answer retrieval, and response retrieval. Specifically, we attempt to study the following two questions: 1) Does relevance modeling in these tasks really show differences in terms of natural language understanding (NLU)? We employ 16 linguistic tasks to probe a unified retrieval model over these three retrieval tasks to answer this question. 2) If there do exist differences, how can we leverage the findings to enhance the relevance modeling? We proposed three intervention methods to investigate how to leverage different modeling focuses of relevance to improve these IR tasks. We believe the way we study the problem as well as our findings would be beneficial to the IR community.