# One-Shot Labeling for Automatic Relevance Estimation

arXiv: https://arxiv.org/abs/2302.11266 · HTML: https://mlsanity.com/paper/2302.11266/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2023.

## Built on by (most cited first, 30 of 31)

- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies](https://mlsanity.com/paper/2309.13063/) (2023)
- [When do Generative Query and Document Expansions Fail? A Comprehensive Study Across Methods, Retrievers, and Datasets](https://mlsanity.com/paper/2309.08541/) (2023)
- [Query Performance Prediction using Relevance Judgments Generated by Large Language Models](https://mlsanity.com/paper/2404.01012/) (2024)
- [Can We Use Large Language Models to Fill Relevance Judgment Holes?](https://mlsanity.com/paper/2405.05600/) (2024)
- [Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation](https://mlsanity.com/paper/2503.19092/) (2025)
- [A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment](https://mlsanity.com/paper/2504.12408/) (2025)
- [Benchmarking LLM-based Relevance Judgment Methods](https://mlsanity.com/paper/2504.12558/) (2025)
- [DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queries](https://mlsanity.com/paper/2310.04678/) (2023)
- [Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I](https://mlsanity.com/paper/2407.02464/) (2024)
- [How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective](https://mlsanity.com/paper/2504.07898/) (2025)
- [Can AI Truly Represent Your Voice in Deliberations? A Comprehensive Study of Large-Scale Opinion Aggregation with LLMs](https://mlsanity.com/paper/2510.05154/) (2025)
- [Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluation](https://mlsanity.com/paper/2408.01363/) (2024)
- [ConQRet: Benchmarking Fine-Grained Evaluation of Retrieval Augmented Argumentation with LLM Judges](https://mlsanity.com/paper/2412.05206/) (2024)
- [Generative Product Recommendations for Implicit Superlative Queries](https://mlsanity.com/paper/2504.18748/) (2025)
- [LLM-based Relevance Assessment for Web-Scale Search Evaluation at Pinterest](https://mlsanity.com/paper/2509.03764/) (2025)
- [Best in Tau@LLMJudge: Criteria-Based Relevance Evaluation with Llama3](https://mlsanity.com/paper/2410.14044/) (2024)
- [Boosting LLM-based Relevance Modeling with Distribution-Aware Robust Learning](https://mlsanity.com/paper/2412.12504/) (2024)
- [Augmented Relevance Datasets with Fine-Tuned Small LLMs](https://mlsanity.com/paper/2504.09816/) (2025)
- [The Viability of Crowdsourcing for RAG Evaluation](https://mlsanity.com/paper/2504.15689/) (2025)
- [RecSys Arena: Pair-wise Recommender System Evaluation with Large Language Models](https://mlsanity.com/paper/2412.11068/) (2024)
- [Fashion-AlterEval: A Dataset for Improved Evaluation of Conversational Recommendation Systems with Alternative Relevant Items](https://mlsanity.com/paper/2507.18017/) (2025)
- [Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessment](https://mlsanity.com/paper/2601.01862/) (2026)
- [Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks](https://mlsanity.com/paper/2602.06526/) (2026)
- [DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation](https://mlsanity.com/paper/2605.04458/) (2026)
- [ADORE: Iterative Query Expansion with Retrieval-Grounded Relevance Feedback](https://mlsanity.com/paper/2606.13905/) (2026)
- [Fine-tuning Small Language Models as Efficient Enterprise Search Relevance Labelers](https://mlsanity.com/paper/2601.03211/) (2026)
- [Re-Rankers as Relevance Judges](https://mlsanity.com/paper/2601.04455/) (2026)
- [Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments](https://mlsanity.com/paper/2602.23234/) (2026)
- [RAQG-QPP: Query Performance Prediction with Retrieved Query Variants and Retrieval Augmented Query Generation](https://mlsanity.com/paper/2604.27244/) (2026)

## Abstract (arXiv)

Dealing with unjudged documents ("holes") in relevance assessments is a perennial problem when evaluating search systems with offline experiments. Holes can reduce the apparent effectiveness of retrieval systems during evaluation and introduce biases in models trained with incomplete data. In this work, we explore whether large language models can help us fill such holes to improve offline evaluations. We examine an extreme, albeit common, evaluation setting wherein only a single known relevant document per query is available for evaluation. We then explore various approaches for predicting the relevance of unjudged documents with respect to a query and the known relevant document, including nearest neighbor, supervised, and prompting techniques. We find that although the predictions of these One-Shot Labelers (1SL) frequently disagree with human assessments, the labels they produce yield a far more reliable ranking of systems than the single labels do alone. Specifically, the strongest approaches can consistently reach system ranking correlations of over 0.86 with the full rankings over a variety of measures. Meanwhile, the approach substantially increases the reliability of t-tests due to filling holes in relevance assessments, giving researchers more confidence in results they find to be significant. Alongside this work, we release an easy-to-use software package to enable the use of 1SL for evaluation of other ad-hoc collections or systems.