# LLMs Can Patch Up Missing Relevance Judgments in Evaluation

arXiv: https://arxiv.org/abs/2405.04727 · HTML: https://mlsanity.com/paper/2405.04727/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2024.

## Built on by (most cited first, 12 of 12)

- [Benchmarking LLM-based Relevance Judgment Methods](https://mlsanity.com/paper/2504.12558/) (2025)
- [Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluation](https://mlsanity.com/paper/2409.11860/) (2024)
- [Zero-Shot Dense Retrieval with Embeddings from Relevance Feedback](https://mlsanity.com/paper/2410.21242/) (2024)
- [LITE: LLM-Impelled efficient Taxonomy Evaluation](https://mlsanity.com/paper/2504.01369/) (2025)
- [Augmented Relevance Datasets with Fine-Tuned Small LLMs](https://mlsanity.com/paper/2504.09816/) (2025)
- [The Viability of Crowdsourcing for RAG Evaluation](https://mlsanity.com/paper/2504.15689/) (2025)
- [How important is Recall for Measuring Retrieval Quality?](https://mlsanity.com/paper/2512.20854/) (2025)
- [SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics](https://mlsanity.com/paper/2606.29894/) (2026)
- [Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation](https://mlsanity.com/paper/2608.10385/) (2026)
- [R3A: Reinforced Reasoning for Relevance Assessment for RAG in User-Generated Content Platforms](https://mlsanity.com/paper/2508.02506/) (2025)
- [Re-Rankers as Relevance Judges](https://mlsanity.com/paper/2601.04455/) (2026)
- [Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines](https://mlsanity.com/paper/2609.24122/) (2026)

## Abstract (arXiv)

Unjudged documents or holes in information retrieval benchmarks are considered non-relevant in evaluation, yielding no gains in measuring effectiveness. However, these missing judgments may inadvertently introduce biases into the evaluation as their prevalence for a retrieval model is heavily contingent on the pooling process. Thus, filling holes becomes crucial in ensuring reliable and accurate evaluation. Collecting human judgment for all documents is cumbersome and impractical. In this paper, we aim at leveraging large language models (LLMs) to automatically label unjudged documents. Our goal is to instruct an LLM using detailed instructions to assign fine-grained relevance judgments to holes. To this end, we systematically simulate scenarios with varying degrees of holes by randomly dropping relevant documents from the relevance judgment in TREC DL tracks. Our experiments reveal a strong correlation between our LLM-based method and ground-truth relevance judgments. Based on our simulation experiments conducted on three TREC DL datasets, in the extreme scenario of retaining only 10% of judgments, our method achieves a Kendall tau correlation of 0.87 and 0.92 on an average for Vicu~na-7B and GPT-3.5 Turbo respectively.