# Evaluating D-MERIT of Partial-annotation on Information Retrieval

arXiv: https://arxiv.org/abs/2406.16048 · HTML: https://mlsanity.com/paper/2406.16048/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2024.

## Built on by (most cited first, 4 of 4)

- [NER Retriever: Zero-Shot Named Entity Retrieval with Type-Aware Embeddings](https://mlsanity.com/paper/2509.04011/) (2025)
- [Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks](https://mlsanity.com/paper/2602.06526/) (2026)
- [Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability](https://mlsanity.com/paper/2608.11238/) (2026)
- [AIANO: Enhancing Information Retrieval with AI-Augmented Annotation](https://mlsanity.com/paper/2602.04579/) (2026)

## Abstract (arXiv)

Retrieval models are often evaluated on partially-annotated datasets. Each query is mapped to a few relevant texts and the remaining corpus is assumed to be irrelevant. As a result, models that successfully retrieve false negatives are punished in evaluation. Unfortunately, completely annotating all texts for every query is not resource efficient. In this work, we show that using partially-annotated datasets in evaluation can paint a distorted picture. We curate D-MERIT, a passage retrieval evaluation set from Wikipedia, aspiring to contain all relevant passages for each query. Queries describe a group (e.g., "journals about linguistics") and relevant passages are evidence that entities belong to the group (e.g., a passage indicating that "Language" is a journal about linguistics). We show that evaluating on a dataset containing annotations for only a subset of the relevant passages might result in misleading ranking of the retrieval systems and that as more relevant texts are included in the evaluation set, the rankings converge. We propose our dataset as a resource for evaluation and our study as a recommendation for balance between resource-efficiency and reliable evaluation when annotating evaluation sets for text retrieval.