mlsanityData to 7 Oct 2026

Agent-as-a-Judge: Evaluate Agents with Agents

arXiv 2410.10934 ↗First posted 14 Oct 2024cs.AIcited by 223

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Contemporary evaluation techniques are inadequate for agentic systems. These approaches either focus exclusively on final outcomes -- ignoring the step-by-step nature of agentic systems, or require excessive manual labour. To address this, we introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We apply the Agent-as-a-Judge to the task of code generation. To overcome issues with existing benchmarks and provide a proof-of-concept testbed for Agent-as-a-Judge, we present DevAI, a new benchmark of 55 realistic automated AI development tasks. It includes rich manual annotations, like a total of 365 hierarchical user requirements. We benchmark three of the popular agentic systems using Agent-as-a-Judge and find it dramatically outperforms LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether, we believe that Agent-as-a-Judge marks a concrete step forward for modern agentic systems -- by providing rich and reliable reward signals necessary for dynamic and scalable self-improvement.

Key facts. Cited by 223 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 80 of its 110 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

80 of 110 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Automated Design of Agentic Systems15 Aug
  2. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents1 Jul
  3. 2024Symbolic Learning Enables Self-Evolving Agents26 Jun
  4. 2024Multi-Agent Collaboration via Cross-Team Orchestration13 Jun
  5. 2024Scaling Large Language Model-based Multi-Agent Collaboration11 Jun
  6. 2024Adaptive In-conversation Team Building for Language Model Agents29 May
  7. 2024Language Agents as Optimizable Graphs26 Feb
  8. 2023Agents: An Open-source Framework for Autonomous Language Agents14 Sep
  9. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug
  10. 2023AgentBench: Evaluating LLMs as Agents7 Aug
  11. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug

Other references · 69

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  12. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
38 more
  1. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  2. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  3. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  4. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  5. 2021Measuring Coding Challenge Competence With APPScited by 879
  6. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  8. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  10. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  11. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  12. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  13. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  14. 2019AutoML: A Survey of the State-of-the-Artcited by 388
  15. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  16. 2023GPTScore: Evaluate as You Desirecited by 357
  17. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  18. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  19. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  20. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  21. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  22. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 222
  23. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  24. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  25. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  26. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  27. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  28. 2024MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolutioncited by 132
  29. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  30. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
  31. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoningcited by 119
  32. 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 119
  33. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  34. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 105
  35. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  36. 2024Can LLM be a Personalized Judge?cited by 99
  37. 2024MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualizationcited by 98
  38. 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 89

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 84; 2026: 132.

223 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  4. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  5. 2025Survey on Evaluation of LLM-based Agentscited by 190
  6. 2025SkillWeaver: Web Agents can Self-Improve by Discovering and Honing Skillscited by 174
  7. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systemscited by 170
  8. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  9. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  10. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  11. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  12. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 104
38 more
  1. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  2. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 94
  3. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 80
  4. 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judgecited by 75
  5. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 57
  6. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  7. 2026AgentRx: Diagnosing AI Agent Failures from Execution Trajectoriescited by 53
  8. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  9. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  10. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 46
  11. 2025Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluationcited by 41
  12. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  13. 2025Who is Introducing the Failure? Automatically Attributing Failures of Multi-Agent Systems via Spectrum Analysiscited by 33
  14. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluationcited by 26
  15. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  16. 2025Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attributioncited by 25
  17. 2026Gym-Anything: Turn any Software into an Agent Environmentcited by 24
  18. 2025DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systemscited by 23
  19. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  20. 2024LLMs Can Simulate Standardized Patients via Agent Coevolutioncited by 22
  21. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  22. 2025Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agentscited by 20
  23. 2025RAFFLES: Reasoning-based Attribution of Faults for LLM Systemscited by 19
  24. 2026Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verificationcited by 19
  25. 2025Automatic Failure Attribution and Critical Step Prediction Method for Multi-Agent Systems Based on Causal Inferencecited by 18
  26. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 18
  27. 2025Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solutioncited by 17
  28. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 17
  29. 2025MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Themcited by 16
  30. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMscited by 16
  31. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  32. 2026The Red Queen G"odel Machine: Co-Evolving Agents and Their Evaluatorscited by 16
  33. 2025Beyond Outlining: Heterogeneous Recursive Planning for Adaptive Long-form Writing with Language Modelscited by 15
  34. 2026Agent-as-a-Judgecited by 15
  35. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  36. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  37. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 13
  38. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12

50 most cited of 223 All 223 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026A Framework for Assessing AI Agent Decisions and Outcomes in AutoML Pipelines
  2. 2026Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
  3. 2026Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
  4. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  5. 2025Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents
  6. 2026AgentBeats: Agentifying Agent Assessment for Openness, Standardization, and Reproducibility
  7. 2026A Framework for Evaluating Agentic Skills at Scale
  8. 2025AutoLibra: Agent Metric Induction from Open-Ended Human Feedback