# How important is Recall for Measuring Retrieval Quality?

arXiv: https://arxiv.org/abs/2512.20854 · HTML: https://mlsanity.com/paper/2512.20854/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 24 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12941/) (2024)
- [A Theoretical Analysis of NDCG Type Ranking Measures](https://mlsanity.com/paper/1304.6480/) (2013)
- [The Distracting Effect: Understanding Irrelevant Passages in RAG](https://mlsanity.com/paper/2505.06914/) (2025)
- [Improving Retrieval Augmented Language Model with Self-Reasoning](https://mlsanity.com/paper/2407.19813/) (2024)
- [Active Testing: An Efficient and Robust Framework for Estimating Accuracy](https://mlsanity.com/paper/1807.00493/) (2018)
- [LLMs Can Patch Up Missing Relevance Judgments in Evaluation](https://mlsanity.com/paper/2405.04727/) (2024)
- [LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help?](https://mlsanity.com/paper/2411.06877/) (2024)
- [Mark-Evaluate: Assessing Language Generation using Population Estimation Methods](https://mlsanity.com/paper/2010.04606/) (2020)
- [Towards Meaningful Statements in IR Evaluation. Mapping Evaluation Measures to Interval Scales](https://mlsanity.com/paper/2101.02668/) (2021)
- [Breaking the Lens of the Telescope: Online Relevance Estimation over Large Retrieval Sets](https://mlsanity.com/paper/2504.09353/) (2025)
- [Effective Data Pruning through Score Extrapolation](https://mlsanity.com/paper/2506.09010/) (2025)
- [Intelligent Topic Selection for Low-Cost Information Retrieval Evaluation: A New Perspective on Deep vs. Shallow Judging](https://mlsanity.com/paper/1701.07810/) (2017)
- [Nearly optimal capture-recapture sampling and empirical likelihood weighting estimation for M-estimation with big data](https://mlsanity.com/paper/2209.04569/) (2022)

## Built on by (most cited first, 1 of 1)

- [When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents](https://mlsanity.com/paper/2606.23937/) (2026)

## Abstract (arXiv)

In realistic retrieval settings with large and evolving knowledge bases, the total number of documents relevant to a query is typically unknown, and recall cannot be computed. In this paper, we evaluate several established strategies for handling this limitation by measuring the correlation between retrieval quality metrics and LLM-based judgments of response quality, where responses are generated from the retrieved documents. We conduct experiments across multiple datasets with a relatively low number of relevant documents (2-15). We also introduce a simple retrieval quality measure that performs well without requiring knowledge of the total number of relevant documents.