# Don't Use LLMs to Make Relevance Judgments

arXiv: https://arxiv.org/abs/2409.15133 · HTML: https://mlsanity.com/paper/2409.15133/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2024.

## Built on by (most cited first, 26 of 26)

- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look](https://mlsanity.com/paper/2411.08275/) (2024)
- [Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation](https://mlsanity.com/paper/2503.19092/) (2025)
- [MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answering](https://mlsanity.com/paper/2505.24040/) (2025)
- [The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corpora](https://mlsanity.com/paper/2507.07543/) (2025)
- [To Judge or not to Judge: Using LLM Judgements for Advertiser Keyphrase Relevance at eBay](https://mlsanity.com/paper/2505.04209/) (2025)
- [The Viability of Crowdsourcing for RAG Evaluation](https://mlsanity.com/paper/2504.15689/) (2025)
- [Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corpora](https://mlsanity.com/paper/2505.08905/) (2025)
- [Expanding Relevance Judgments for Medical Case-based Retrieval Task with Multimodal LLMs](https://mlsanity.com/paper/2506.17782/) (2025)
- [LLMDistill4Ads: Using Cross-Encoders to Distill from LLM Signals for Advertiser Keyphrase Recommendations at eBay](https://mlsanity.com/paper/2508.03628/) (2025)
- [Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusing](https://mlsanity.com/paper/2512.00390/) (2025)
- [Succeeding at Scale: Enterprise Retrieval Benchmark Construction and Index-Preserving Query Adaptation for Multi-Tenant Search](https://mlsanity.com/paper/2601.04646/) (2026)
- [Query-Document Dense Vectors for LLM Relevance Judgment Bias Analysis](https://mlsanity.com/paper/2601.01751/) (2026)
- [Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessment](https://mlsanity.com/paper/2601.01862/) (2026)
- [Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?](https://mlsanity.com/paper/2601.13227/) (2026)
- [Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarks](https://mlsanity.com/paper/2602.06526/) (2026)
- [Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation](https://mlsanity.com/paper/2608.10385/) (2026)
- [REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resources](https://mlsanity.com/paper/2504.07584/) (2025)
- [When LLMs Disagree: Diagnosing Relevance Filtering Bias and Retrieval Divergence in SDG Search](https://mlsanity.com/paper/2507.02139/) (2025)
- [SIGIR 2025 -- LiveRAG Challenge Report](https://mlsanity.com/paper/2507.04942/) (2025)
- [The Effect of Document Summarization on LLM-Based Relevance Judgments](https://mlsanity.com/paper/2512.05334/) (2025)
- [Re-Rankers as Relevance Judges](https://mlsanity.com/paper/2601.04455/) (2026)
- [Plans for Evaluating Structured Generative Search Summaries](https://mlsanity.com/paper/2605.26400/) (2026)
- [Retrieval, not hallucinations, will be the limiting factor for LLM-based clinical AI tools](https://mlsanity.com/paper/2607.24793/) (2026)
- [Dear Algo: A Precision-First Agentic Intent Layer for Unified Search and Recommendation](https://mlsanity.com/paper/2608.15877/) (2026)
- [Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines](https://mlsanity.com/paper/2609.24122/) (2026)

## Abstract (arXiv)

Making the relevance judgments for a TREC-style test collection can be complex and expensive. A typical TREC track usually involves a team of six contractors working for 2-4 weeks. Those contractors need to be trained and monitored. Software has to be written to support recording relevance judgments correctly and efficiently. The recent advent of large language models that produce astoundingly human-like flowing text output in response to a natural language prompt has inspired IR researchers to wonder how those models might be used in the relevance judgment collection process. At the ACM SIGIR 2024 conference, a workshop ``LLM4Eval'' provided a venue for this work, and featured a data challenge activity where participants reproduced TREC deep learning track judgments, as was done by Thomas et al (arXiv:2408.08896, arXiv:2309.10621). I was asked to give a keynote at the workshop, and this paper presents that keynote in article form. The bottom-line-up-front message is, don't use LLMs to create relevance judgments for TREC-style evaluations.