# A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look

arXiv: https://arxiv.org/abs/2411.08275 · HTML: https://mlsanity.com/paper/2411.08275/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 13 Nov 2024.

## Builds on (most cited first, 12 of 12)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [LM vs LM: Detecting Factual Errors via Cross Examination](https://mlsanity.com/paper/2305.13281/) (2023)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023)
- [AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Prediction](https://mlsanity.com/paper/2305.09620/) (2023)
- [UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor](https://mlsanity.com/paper/2406.06519/) (2024)
- [Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track](https://mlsanity.com/paper/2406.16828/) (2024)
- [Language Models Trained on Media Diets Can Predict Public Opinion](https://mlsanity.com/paper/2303.16779/) (2023)
- [Don't Use LLMs to Make Relevance Judgments](https://mlsanity.com/paper/2409.15133/) (2024)
- [A Comparison of Methods for Evaluating Generative IR](https://mlsanity.com/paper/2404.04044/) (2024)

## Built on by (most cited first, 18 of 18)

- [DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis](https://mlsanity.com/paper/2508.20033/) (2025)
- [Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework](https://mlsanity.com/paper/2411.09607/) (2024)
- [The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models](https://mlsanity.com/paper/2504.15068/) (2025)
- [Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation](https://mlsanity.com/paper/2503.19092/) (2025)
- [A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment](https://mlsanity.com/paper/2504.12408/) (2025)
- [FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents](https://mlsanity.com/paper/2504.13128/) (2025)
- [Benchmarking LLM-based Relevance Judgment Methods](https://mlsanity.com/paper/2504.12558/) (2025)
- [Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges](https://mlsanity.com/paper/2504.15205/) (2025)
- [CliniQ: A Multi-faceted Benchmark for Electronic Health Record Retrieval with Semantic Match Assessment](https://mlsanity.com/paper/2502.06252/) (2025)
- [The Viability of Crowdsourcing for RAG Evaluation](https://mlsanity.com/paper/2504.15689/) (2025)
- [Rethinking Evidence Hierarchies in Medical Language Benchmarks: A Critical Evaluation of HealthBench](https://mlsanity.com/paper/2508.00081/) (2025)
- [Incorporating Q&A Nuggets into Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.13222/) (2026)
- [Plans for Evaluating Structured Generative Search Summaries](https://mlsanity.com/paper/2605.26400/) (2026)
- [Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs](https://mlsanity.com/paper/2606.17634/) (2026)
- [HIERA: Hierarchical Multi-Agent Relevance Assessment for Content Discovery Systems](https://mlsanity.com/paper/2608.00785/) (2026)
- [Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selection](https://mlsanity.com/paper/2609.02745/) (2026)
- [Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems](https://mlsanity.com/paper/2609.08887/) (2026)
- [Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines](https://mlsanity.com/paper/2609.24122/) (2026)

## Abstract (arXiv)

The application of large language models to provide relevance assessments presents exciting opportunities to advance information retrieval, natural language processing, and beyond, but to date many unknowns remain. This paper reports on the results of a large-scale evaluation (the TREC 2024 RAG Track) where four different relevance assessment approaches were deployed in situ: the "standard" fully manual process that NIST has implemented for decades and three different alternatives that take advantage of LLMs to different extents using the open-source UMBRELA tool. This setup allows us to correlate system rankings induced by the different approaches to characterize tradeoffs between cost and quality. We find that in terms of nDCG@20, nDCG@100, and Recall@100, system rankings induced by automatically generated relevance assessments from UMBRELA correlate highly with those induced by fully manual assessments across a diverse set of 77 runs from 19 teams. Our results suggest that automatically generated UMBRELA judgments can replace fully manual judgments to accurately capture run-level effectiveness. Surprisingly, we find that LLM assistance does not appear to increase correlation with fully manual assessments, suggesting that costs associated with human-in-the-loop processes do not bring obvious tangible benefits. Overall, human assessors appear to be stricter than UMBRELA in applying relevance criteria. Our work validates the use of LLMs in academic TREC-style evaluations and provides the foundation for future studies.