mlsanityData to 7 Oct 2026

Agent-as-a-Judge

arXiv 2601.05111 ↗First posted 8 Jan 2026cs.CL cs.AIcited by 15

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

LLM-as-a-Judge has revolutionized AI evaluation by leveraging large language models for scalable assessments. However, as evaluands become increasingly complex, specialized, and multi-step, the reliability of LLM-as-a-Judge has become constrained by inherent biases, shallow single-pass reasoning, and the inability to verify assessments against real-world observations. This has catalyzed the transition to Agent-as-a-Judge, where agentic judges employ planning, tool-augmented verification, multi-agent collaboration, and persistent memory to enable more robust, verifiable, and nuanced evaluations. Despite the rapid proliferation of agentic evaluation systems, the field lacks a unified framework to navigate this shifting landscape. To bridge this gap, we present the first comprehensive survey tracing this evolution. Specifically, we identify key dimensions that characterize this paradigm shift and establish a developmental taxonomy. We organize core methodologies and survey applications across general and professional domains. Furthermore, we analyze frontier challenges and identify promising research directions, ultimately providing a clear roadmap for the next generation of agentic evaluation.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 52 of its 62 references that are in the map. First posted on arXiv 8 Jan 2026.

Builds on

52 of 62 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition26 Sep
  2. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence28 Jul
  3. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation28 Jul
  4. 2025HiMATE: A Hierarchical Multi-Agent Framework for Machine Translation Evaluation22 May
  5. 2025CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring20 May
  6. 2025Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications1 Apr
  7. 2025VerifiAgent: a Unified Verification Agent in Language Model Reasoning1 Apr
  8. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifiers27 Feb
  9. 2025IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systems19 Jan
  10. 2024M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluation28 Dec
  11. 2024SAGEval: The frontiers of Satisfactory Agent based NLG Evaluation for reference-free open-ended text25 Nov
  12. 2024Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems15 Nov
4 more
  1. 2024Agent-as-a-Judge: Evaluate Agents with Agents14 Oct
  2. 2024MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation28 Mar
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep
  4. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug

Other references · 36

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  5. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  6. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  7. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  8. 2024AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulatorcited by 99
  9. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62
  10. 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluationscited by 53
  11. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  12. 2024Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Modelscited by 41
24 more
  1. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 41
  2. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 40
  3. 2024AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentationcited by 39
  4. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Rewardcited by 39
  5. 2025SynthesizeMe! Inducing Persona-Guided Prompts for Personalized Reward Models in LLMscited by 26
  6. 2025Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignmentcited by 22
  7. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  8. 2025FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Modelscited by 19
  9. 2024Grade Like a Human: Rethinking Automated Assessment with Large Language Modelscited by 16
  10. 2024A LLM-Powered Automatic Grading Framework with Human-Level Guidelines Optimizationcited by 16
  11. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  12. 2025A Unified Agentic Framework for Evaluating Conditional Image Generationcited by 15
  13. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  14. 2025FinDeepResearch: Evaluating Deep Research Agents in Rigorous Financial Analysiscited by 13
  15. 2025EvalAgent: Discovering Implicit Evaluation Criteria from the Webcited by 11
  16. 2025FinResearchBench: A Logic Tree based Agent-as-a-Judge Evaluation Framework for Financial Research Agentscited by 11
  17. 2025Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluationcited by 10
  18. 2025PSYCHE: A Multi-faceted Patient Simulation Framework for Evaluation of Psychiatric Assessment Conversational Agentscited by 9
  19. 2025Agent-as-Judge for Factual Summarization of Long Narrativescited by 9
  20. 2025ESC-Judge: A Framework for Comparing Emotional Support Conversational Agentscited by 8
  21. 2025CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generationcited by 6
  22. 2025Self-Adaptive Cognitive Debiasing for Large Language Models in Decision-Makingcited by 4
  23. 2025UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarkingcited by 4
  24. 2025SAMVAD: A Multi-Agent System for Simulating Judicial Deliberation Dynamics in Indiacited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Taskscited by 41
  2. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 17
  3. 2026HarnessEval-W: Agentifying the Evaluation of Visual Worldscited by 3
  4. 2026Scaling Medical Reasoning Verification via Tool-Integrated Reinforcement Learningcited by 2
  5. 2026Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Modelscited by 2
  6. 2026Self-Distilled Reinforcement Learning for Co-Evolving Agentic Recommender Systemscited by 1
  7. 2026Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrievalcited by 1
  8. 2026SetupX: Can LLM Agents Learn from Past Failures in Functionality-Correct Code Repository Setup?cited by 1
  9. 2026ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Taskscited by 1
  10. 2026StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflowscited by 1
  11. 2026Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluationcited by 0
  12. 2026Synthetic Scenario Generation for Evaluation of Industry 4.0 Agentscited by 0
3 more
  1. 2026Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluationcited by 0
  2. 2026Learning Strategies to Break Judgescited by 0
  3. 2026Maat: Independent Deterministic Contract-Based Governance for Multi-Agent LLM Workflowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
  2. 2025Evaluation and Benchmarking of LLM Agents: A Survey
  3. 2025Survey on Evaluation of LLM-based Agents
  4. 2025Auto-Eval Judge: Towards a General Agentic Framework for Task Completion Evaluation
  5. 2026Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
  6. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  7. 2026CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration
  8. 2026AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation