mlsanityData to 7 Oct 2026

Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation

arXiv 2507.21028 ↗First posted 28 Jul 2025cs.CLcited by 26

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Nearly all human work is collaborative; thus, the evaluation of real-world NLP applications often requires multiple dimensions that align with diverse human perspectives. As real human evaluator resources are often scarce and costly, the emerging "LLM-as-a-judge" paradigm sheds light on a promising approach to leverage LLM agents to believably simulate human evaluators. Yet, to date, existing LLM-as-a-judge approaches face two limitations: persona descriptions of agents are often arbitrarily designed, and the frameworks are not generalizable to other tasks. To address these challenges, we propose MAJ-EVAL, a Multi-Agent-as-Judge evaluation framework that can automatically construct multiple evaluator personas with distinct dimensions from relevant text documents (e.g., research papers), instantiate LLM agents with the personas, and engage in-group debates with multi-agents to Generate multi-dimensional feedback. Our evaluation experiments in both the educational and medical domains demonstrate that MAJ-EVAL can generate evaluation results that better align with human experts' ratings compared with conventional automated evaluation metrics and existing LLM-as-a-judge methods.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 41 of its 72 references that are in the map. First posted on arXiv 28 Jul 2025.

Builds on

41 of 72 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Agent-as-a-Judge: Evaluate Agents with Agents14 Oct
  2. 2024MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation28 Mar
  3. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors21 Aug

Other references · 38

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  8. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  9. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  10. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  11. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  12. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
26 more
  1. 2024Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Taskscited by 102
  2. 2021MS2: Multi-Document Summarization of Medical Studiescited by 80
  3. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  4. 2022Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehensioncited by 79
  5. 2025LLM Generated Persona is a Promise with a Catchcited by 77
  6. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluatorscited by 42
  7. 2024PRE: A Peer Review Based Large Language Model Evaluatorcited by 28
  8. 2022Educational Question Generation of Children Storybooks via Question Type Distribution Learning and Event-Centric Summarizationcited by 26
  9. 2025Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agentscited by 22
  10. 2023MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluationcited by 19
  11. 2025Ask Patients with Patience: Enabling LLMs for Human-Centric Medical Dialogue with Grounded Reasoningcited by 19
  12. 2023Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviewscited by 18
  13. 2023Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluationscited by 18
  14. 2024DEBATE: Devil's Advocate-Based Assessment and Text Evaluationcited by 16
  15. 2023Are Human Explanations Always Helpful? Towards Objective Evaluation of Human Natural Language Explanationscited by 13
  16. 2023More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineeringcited by 13
  17. 2023Human Still Wins over LLM: An Empirical Study of Active Learning on Domain-Specific Annotation Taskscited by 13
  18. 2025BookWorld: From Novels to Interactive Agent Societies for Creative Story Generationcited by 12
  19. 2023Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecturecited by 10
  20. 2025Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluationcited by 10
  21. 2025UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agentscited by 10
  22. 2024Exploring Parent's Needs for Children-Centered AI to Support Preschoolers' Interactive Storytelling and Reading Activitiescited by 8
  23. 2023StorySparkQA: Expert-Annotated QA Pairs with Real-World Knowledge for Children's Story-Based Learningcited by 4
  24. 2025Characterizing LLM-Empowered Personalized Story-Reading and Interaction for Children: Insights from Multi-Stakeholder Perspectivescited by 4
  25. 2024Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Reviewcited by 3
  26. 2025RECOVER: Designing a Large Language Model-based Remote Patient Monitoring System for Postoperative Gastrointestinal Cancer Carecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 18.

26 papers.

Sort:

By area:

  1. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  2. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMscited by 16
  3. 2025LLM Agent Meets Agentic AI: Can LLM Agents Simulate Customers to Evaluate Agentic-AI-based Shopping Assistants?cited by 15
  4. 2026Agent-as-a-Judgecited by 15
  5. 2026Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learningcited by 9
  6. 2025Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systemscited by 6
  7. 2025DPRF: A Generalizable Dynamic Persona Refinement Framework for Optimizing Behavior Alignment Between Personalized LLM Role-Playing Agents and Humanscited by 5
  8. 2025Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaborationcited by 4
  9. 2025Is Your Prompt Poisoning Code? Defect Induction Rates and Security Mitigation Strategiescited by 3
  10. 2026Meta-Judging with Large Language Models: Concepts, Methods, and Challengescited by 3
  11. 2026LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise Taskscited by 3
  12. 2026From Human-Human Collaboration to Human-Agent Collaboration: A Vision, Design Philosophy, and an Empirical Framework for Achieving Successful Partnerships Between Humans and LLM Agentscited by 2
14 more
  1. 2026When Verification Hurts: Asymmetric Effects of Multi-Agent Feedback in Logic Proof Tutoringcited by 2
  2. 2026ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluationcited by 2
  3. 2026CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experimentscited by 2
  4. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  5. 2026Redundant or Necessary? A Benchmark for Detecting Redundant Steps in Agent Trajectoriescited by 1
  6. 2026The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judgecited by 1
  7. 2026Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debatecited by 1
  8. 2025Approximating Human Preferences Using a Multi-Judge Learned Systemcited by 0
  9. 2026SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systemscited by 0
  10. 2026An Empirical Study of Automating Agent Evaluationcited by 0
  11. 2026Self-Refining Topology Optimization via an LLM-Based Multi-Agent Frameworkcited by 0
  12. 2026CRAFT: Learn the Schema, Execute the Plancited by 0
  13. 2026Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Tracescited by 0
  14. 2026Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
  2. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  3. 2025Multi-Agent LLM Judge: automatic personalized LLM judge design for evaluating natural language generation applications
  4. 2026CollabEval: Enhancing LLM-as-a-Judge via Multi-Agent Collaboration
  5. 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation
  6. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  7. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions
  8. 2024Aligning Human and LLM Judgments: Insights from EvalAssist on Task-Specific Evaluations and AI-assisted Assessment Strategy Preferences