mlsanityData to 7 Oct 2026

Don't Use LLMs to Make Relevance Judgments

arXiv 2409.15133 ↗First posted 23 Sep 2024cs.IRcited by 26

Not on the map. Why?

Abstract (arXiv)

Making the relevance judgments for a TREC-style test collection can be complex and expensive. A typical TREC track usually involves a team of six contractors working for 2-4 weeks. Those contractors need to be trained and monitored. Software has to be written to support recording relevance judgments correctly and efficiently. The recent advent of large language models that produce astoundingly human-like flowing text output in response to a natural language prompt has inspired IR researchers to wonder how those models might be used in the relevance judgment collection process. At the ACM SIGIR 2024 conference, a workshop ``LLM4Eval'' provided a venue for this work, and featured a data challenge activity where participants reproduced TREC deep learning track judgments, as was done by Thomas et al (arXiv:2408.08896, arXiv:2309.10621). I was asked to give a keynote at the workshop, and this paper presents that keynote in article form. The bottom-line-up-front message is, don't use LLMs to create relevance judgments for TREC-style evaluations.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Sep 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 11.

26 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Lookcited by 18
  3. 2025Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluationcited by 15
  4. 2025MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answeringcited by 5
  5. 2025The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corporacited by 4
  6. 2025To Judge or not to Judge: Using LLM Judgements for Advertiser Keyphrase Relevance at eBaycited by 3
  7. 2025The Viability of Crowdsourcing for RAG Evaluationcited by 2
  8. 2025Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corporacited by 2
  9. 2025Expanding Relevance Judgments for Medical Case-based Retrieval Task with Multimodal LLMscited by 2
  10. 2025LLMDistill4Ads: Using Cross-Encoders to Distill from LLM Signals for Advertiser Keyphrase Recommendations at eBaycited by 2
  11. 2025Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusingcited by 2
  12. 2026Query-Document Dense Vectors for LLM Relevance Judgment Bias Analysiscited by 1
14 more
  1. 2026Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessmentcited by 1
  2. 2026Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?cited by 1
  3. 2026Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarkscited by 1
  4. 2025REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resourcescited by 0
  5. 2025When LLMs Disagree: Diagnosing Relevance Filtering Bias and Retrieval Divergence in SDG Searchcited by 0
  6. 2025SIGIR 2025 -- LiveRAG Challenge Reportcited by 0
  7. 2025The Effect of Document Summarization on LLM-Based Relevance Judgmentscited by 0
  8. 2026Re-Rankers as Relevance Judgescited by 0
  9. 2026Succeeding at Scale: Enterprise Retrieval Benchmark Construction and Index-Preserving Query Adaptation for Multi-Tenant Searchcited by 0
  10. 2026Plans for Evaluating Structured Generative Search Summariescited by 0
  11. 2026Retrieval, not hallucinations, will be the limiting factor for LLM-based clinical AI toolscited by 0
  12. 2026Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluationcited by 0
  13. 2026Dear Algo: A Precision-First Agentic Intent Layer for Unified Search and Recommendationcited by 0
  14. 2026Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelinescited by 0