mlsanityData to 14 Oct 2026

Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models

arXiv 2404.18796 ↗First posted 29 Apr 2024cs.CL cs.AIcited by 326

Not on the map. Why?

Cited by 51 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As Large Language Models (LLMs) have become more advanced, they have outpaced our abilities to accurately evaluate their quality. Not only is finding data to adequately probe particular model properties difficult, but evaluating the correctness of a model's freeform generation alone is a challenge. To address this, many evaluations now rely on using LLMs themselves as judges to score the quality of outputs from other LLMs. Evaluations most commonly use a single large model like GPT4. While this method has grown in popularity, it is costly, has been shown to introduce intramodel bias, and in this work, we find that very large models are often unnecessary. We propose instead to evaluate models using a Panel of LLm evaluators (PoLL). Across three distinct judge settings and spanning six different datasets, we find that using a PoLL composed of a larger number of smaller models outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families, and does so while being over seven times less expensive.

Key facts. Cited by 326 papers in the map (data through 14 Oct 2026). Builds on 17 of its 33 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

17 of 33 references.

References · 17

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  11. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  12. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
5 more
  1. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  2. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  3. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  4. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  5. 2022Multi-hop Question Answeringcited by 64

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 46; 2025: 90; 2026: 190.

326 papers.

Sort:

By area:

  1. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  4. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  5. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  6. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  7. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  8. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 117
  9. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  10. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  11. 2024Can LLM be a Personalized Judge?cited by 100
  12. 2024On scalable oversight with weak LLMs judging strong LLMscited by 99
38 more
  1. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  2. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 85
  3. 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
  4. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  5. 2025MedHELM: Holistic Evaluation of Large Language Models for Medical Taskscited by 64
  6. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 54
  7. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  8. 2025Correlated Errors in Large Language Modelscited by 45
  9. 2024Improving Reward Models with Synthetic Critiquescited by 44
  10. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 43
  11. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longercited by 41
  12. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
  13. 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 38
  14. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  15. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 36
  16. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35
  17. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 35
  18. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
  19. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  20. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  21. 2024Constructing Domain-Specific Evaluation Sets for LLM-as-a-judgecited by 30
  22. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 30
  23. 2024Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Contextcited by 28
  24. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 28
  25. 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifactscited by 26
  26. 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 26
  27. 2024Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QAcited by 25
  28. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 25
  29. 2025Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settingscited by 24
  30. 2025Shrinking the Generation-Verification Gap with Weak Verifierscited by 24
  31. 2025RAFFLES: Reasoning-based Attribution of Faults for LLM Systemscited by 21
  32. 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 19
  33. 2024CrossCheckGPT: Universal Hallucination Ranking for Multimodal Foundation Modelscited by 18
  34. 2024Black-box Uncertainty Quantification Method for LLM-as-a-Judgecited by 18
  35. 2025Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorerscited by 18
  36. 2024"Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?cited by 17
  37. 2026Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptionscited by 17
  38. 2024Training Language Models to Critique With Multi-agent Feedbackcited by 16

50 most cited of 326 All 326 (JSON)