mlsanityData to 14 Oct 2026

When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs

arXiv 2508.02994 ↗First posted 5 Aug 2025cs.AIcited by 17

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

As large language models (LLMs) grow in capability and autonomy, evaluating their outputs-especially in open-ended and complex tasks-has become a critical bottleneck. A new paradigm is emerging: using AI agents as the evaluators themselves. This "agent-as-a-judge" approach leverages the reasoning and perspective-taking abilities of LLMs to assess the quality and safety of other models, promising calable and nuanced alternatives to human evaluation. In this review, we define the agent-as-a-judge concept, trace its evolution from single-model judges to dynamic multi-agent debate frameworks, and critically examine their strengths and shortcomings. We compare these approaches across reliability, cost, and human alignment, and survey real-world deployments in domains such as medicine, law, finance, and education. Finally, we highlight pressing challenges-including bias, robustness, and meta evaluation-and outline future research directions. By bringing together these strands, our review demonstrates how agent-based judging can complement (but not replace) human oversight, marking a step toward trustworthy, scalable evaluation for next-generation LLMs.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 38 of its 58 references that are in the map. First posted on arXiv 5 Aug 2025.

Builds on

38 of 58 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Evaluation and Benchmarking of LLM Agents: A Survey29 Jul
  2. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation28 Jul
  3. 2025Small Language Models are the Future of Agentic AI2 Jun
  4. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture21 Nov
  5. 2024Agent-as-a-Judge: Evaluate Agents with Agents14 Oct
  6. 2024Mixture-of-Agents Enhances Large Language Model Capabilities7 Jun
  7. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  5. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  7. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  8. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  9. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  10. 2023Detecting Language Model Attacks with Perplexitycited by 431
  11. 2018AI safety via debatecited by 400
  12. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
19 more
  1. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  2. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  3. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 282
  4. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  5. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  6. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 126
  7. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  8. 2023Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversationscited by 89
  9. 2025LLM Generated Persona is a Promise with a Catchcited by 81
  10. 2021MS2: Multi-Document Summarization of Medical Studiescited by 80
  11. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
  12. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 44
  13. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  14. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolutioncited by 42
  15. 2024AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentationcited by 40
  16. 2023MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluationcited by 19
  17. 2024DEBATE: Devil's Advocate-Based Assessment and Text Evaluationcited by 16
  18. 2023StorySparkQA: Expert-Annotated QA Pairs with Real-World Knowledge for Children's Story-Based Learningcited by 4
  19. 2024Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Reviewcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 15.

17 papers.

Sort:

By area:

  1. 2025ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaborationcited by 11
  2. 2026Why Do LLM-based Web Agents Fail? A Hierarchical Planning Perspectivecited by 9
  3. 2026Qworld: Question-Specific Evaluation Criteria for LLMscited by 8
  4. 2026EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repaircited by 6
  5. 2026Security in LLM-as-a-Judge: A Comprehensive SoKcited by 5
  6. 2025Testing the Testers: Human-Driven Quality Assessment of Voice AI Testing Platformscited by 1
  7. 2026Project Synapse: A Hierarchical Multi-Agent Framework with Hybrid Memory for Autonomous Resolution of Last-Mile Delivery Disruptionscited by 1
  8. 2026The Stability Trap: Evaluating the Reliability of LLM-Based Instruction Adherence Auditingcited by 1
  9. 2026AREG: Adversarial Resource Extraction Game for Evaluating Persuasion and Resistance in Large Language Modelscited by 1
  10. 2026PlotTwist: A Creative Plot Generation Framework with Small Language Modelscited by 0
  11. 2026Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluationcited by 0
  12. 2026Semantic Reranking at Inference Time for Hard Examples in Rhetorical Role Labelingcited by 0
5 more
  1. 2026Power Systems Agent Benchmark: Executable Evaluation of AI Agents in Electric Power Engineeringcited by 0
  2. 2026Eta Given Delta: Defining LLM Tool Efficiency With Marginal Tool Utilitycited by 0
  3. 2026Automatic Hard Example Synthesis with Multi-Level Agentic Data Curationcited by 0
  4. 2026How to Dogfood Your AI Chat Agent: A Three-Layer Evaluation Framework with Goal-Directed NPC Simulationcited by 0
  5. 2026MetaSteer: Context-Conditioned, nonlinear Steering via Attention-Projection Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Agent-as-a-Judge
  2. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  3. 2026CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration
  4. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  5. 2026LLM Agents in Law: Taxonomy, Applications, and Challenges
  6. 2025Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications
  7. 2023AgentBench: Evaluating LLMs as Agents
  8. 2025Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judge