mlsanityData to 14 Oct 2026

A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look

arXiv 2411.08275 ↗First posted 13 Nov 2024cs.IR cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

The application of large language models to provide relevance assessments presents exciting opportunities to advance information retrieval, natural language processing, and beyond, but to date many unknowns remain. This paper reports on the results of a large-scale evaluation (the TREC 2024 RAG Track) where four different relevance assessment approaches were deployed in situ: the "standard" fully manual process that NIST has implemented for decades and three different alternatives that take advantage of LLMs to different extents using the open-source UMBRELA tool. This setup allows us to correlate system rankings induced by the different approaches to characterize tradeoffs between cost and quality. We find that in terms of nDCG@20, nDCG@100, and Recall@100, system rankings induced by automatically generated relevance assessments from UMBRELA correlate highly with those induced by fully manual assessments across a diverse set of 77 runs from 19 teams. Our results suggest that automatically generated UMBRELA judgments can replace fully manual judgments to accurately capture run-level effectiveness. Surprisingly, we find that LLM assistance does not appear to increase correlation with fully manual assessments, suggesting that costs associated with human-in-the-loop processes do not bring obvious tangible benefits. Overall, human assessors appear to be stricter than UMBRELA in applying relevance criteria. Our work validates the use of LLMs in academic TREC-style evaluations and provides the foundation for future studies.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 12 of its 32 references that are in the map. First posted on arXiv 13 Nov 2024.

Builds on

12 of 32 references.

References · 12

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  3. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 235
  4. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  5. 2023Large language models can accurately predict searcher preferencescited by 131
  6. 2023Perspectives on Large Language Models for Relevance Judgmentcited by 85
  7. 2023AI-Augmented Surveys: Leveraging Large Language Models and Surveys for Opinion Predictioncited by 48
  8. 2024UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessorcited by 42
  9. 2024Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Trackcited by 32
  10. 2023Language Models Trained on Media Diets Can Predict Public Opinioncited by 30
  11. 2024Don't Use LLMs to Make Relevance Judgmentscited by 26
  12. 2024A Comparison of Methods for Evaluating Generative IRcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 7.

18 papers.

Sort:

By area:

  1. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 34
  2. 2024Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Frameworkcited by 28
  3. 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
  4. 2025Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluationcited by 15
  5. 2025A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgmentcited by 15
  6. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  7. 2025Benchmarking LLM-based Relevance Judgment Methodscited by 14
  8. 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judgescited by 14
  9. 2025CliniQ: A Multi-faceted Benchmark for Electronic Health Record Retrieval with Semantic Match Assessmentcited by 3
  10. 2025The Viability of Crowdsourcing for RAG Evaluationcited by 2
  11. 2025Rethinking Evidence Hierarchies in Medical Language Benchmarks: A Critical Evaluation of HealthBenchcited by 2
  12. 2026Incorporating Q&A Nuggets into Retrieval-Augmented Generationcited by 2
6 more
  1. 2026Plans for Evaluating Structured Generative Search Summariescited by 0
  2. 2026Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphscited by 0
  3. 2026HIERA: Hierarchical Multi-Agent Relevance Assessment for Content Discovery Systemscited by 0
  4. 2026Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selectioncited by 0
  5. 2026Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systemscited by 0
  6. 2026Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelinescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective
  2. 2026PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimation
  3. 2024Evaluating Cost-Accuracy Trade-offs in Multimodal Search Relevance Judgements
  4. 2023"Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation
  5. 2026Why Large Language Models can Secretly Outperform Embedding Similarity in Information Retrieval ↗
  6. 2026SCORE: Specificity, Context Utilization, Robustness, and Relevance for Reference-Free LLM Evaluation
  7. 2026Beyond Relevance: Utility-Centric Retrieval in the LLM Era
  8. 2025Redefining Retrieval Evaluation in the Era of LLMs