mlsanityData to 14 Oct 2026

Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions

arXiv 2405.20267 ↗First posted 30 May 2024cs.CLcited by 42

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

As LLMs continuously evolve, there is an urgent need for a reliable evaluation method that delivers trustworthy results promptly. Currently, static benchmarks suffer from inflexibility and unreliability, leading users to prefer human voting platforms like Chatbot Arena. However, human evaluations require significant manual effort. To address this, we propose the Auto-Arena, an innovative framework that automates the entire evaluation process using LLM-powered agents. Firstly, an LLM examiner generates questions. Then, two LLM candidates engage in a multi-round peer battle based on individual questions, aiming at revealing their true performance differences. Finally, a committee of LLM judges collaboratively discusses and decides the winner, reducing bias and enhancing fairness. During the peer battles, we observe intriguing scenarios where the LLM candidates display competitive behaviors and even learn from the opponents. In our extensive experiments involving 15 recent LLMs, Auto-Arena shows a 92.14% correlation with human preferences, surpassing all previous expert-annotated benchmarks without any manual efforts. As a result, Auto-Arena offers a promising alternative to current human evaluation platforms for evaluating LLMs automatically.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 38 of its 55 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

38 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate14 Aug

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
25 more
  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  5. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  8. 2024Yi: Open Foundation Models by 01.AIcited by 719
  9. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  10. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  11. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  12. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  13. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  14. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  15. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 101
  16. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 88
  17. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  18. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  19. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62
  20. 2023Elo Uncovered: Robustness and Best Practices in Language Model Evaluationcited by 61
  21. 2023LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Constructioncited by 58
  22. 2023An Open Source Data Contamination Report for Large Language Modelscited by 51
  23. 2024KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Modelscited by 36
  24. 2024PRE: A Peer Review Based Large Language Model Evaluatorcited by 28
  25. 2024AgentGroupChat: An Interactive Group Chat Simulacra For Better Eliciting Emergent Behaviorcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 16; 2026: 21.

42 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2024Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agentscited by 97
  4. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  5. 2024GameArena: Evaluating LLM Reasoning through Live Computer Gamescited by 32
  6. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  7. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26
  8. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 13
  9. 2026MoltNet: Understanding Social Behavior of AI Agents in the Agent-Native MoltBookcited by 12
  10. 2026Benchmarks Saturate When The Model Gets Smarter Than The Judgecited by 11
  11. 2025AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agentscited by 9
  12. 2025One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Frameworkcited by 9
30 more
  1. 2025Auto-Eval Judge: Towards a General Agentic Framework for Task Completion Evaluationcited by 8
  2. 2025Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarkscited by 7
  3. 2025AgentBay: A Hybrid Interaction Sandbox for Seamless Human-AI Intervention in Agentic Systemscited by 6
  4. 2026DR-Arena: an Automated Evaluation Framework for Deep Research Agentscited by 6
  5. 2025EvoEmpirBench: Dynamic Spatial Reasoning with Agent-ExpVercited by 5
  6. 2025Efficient MAP Estimation of LLM Judgment Performance with Prior Transfercited by 4
  7. 2026GenArena: How Can We Achieve Human-Aligned Evaluation for Visual Generation Tasks?cited by 4
  8. 2026Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judgecited by 3
  9. 2026Multi-Agent Debate Strategies: Survey, Taxonomy, and Challengescited by 3
  10. 2024CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarizationcited by 2
  11. 2025SCAN: Structured Capability Assessment and Navigation for LLMscited by 2
  12. 2026ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluationcited by 2
  13. 2026Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attributioncited by 2
  14. 2026When Languages Disagree: Self-Evolving Multilingual LLM Judgescited by 2
  15. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluationcited by 2
  16. 2025ReviewInstruct: A Review-Driven Multi-Turn Conversations Generation Method for Large Language Modelscited by 1
  17. 2025AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessmentcited by 1
  18. 2025RevoNAD: Reflective Evolutionary Exploration for Neural Architecture Designcited by 1
  19. 2026StressEval: Failure-Driven Dynamic Benchmarking for Knowledge-Intensive Reasoning in Large Language Modelscited by 1
  20. 2026LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluationcited by 1
  21. 2026Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessmentcited by 1
  22. 2025The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluationscited by 0
  23. 2026MILE-RefHumEval: A Reference-Free, Multi-Independent LLM Framework for Human-Aligned Evaluationcited by 0
  24. 2026Measuring Intelligence Beyond Human Scalecited by 0
  25. 2026Digital Pantheon: Simulating and Auditing Coalition Formation with LLM Agentscited by 0
  26. 2026Agentic Harnesses: LLM-Driven Verification Layers for Robot Autonomycited by 0
  27. 2026Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialoguecited by 0
  28. 2026You Really Didn't Get That? Benchmarking Social Pragmatic Inference for Indirect and Playful Chinese Online Commentscited by 0
  29. 2026DUET: Co-Evolving Solver and Grader Agentscited by 0
  30. 2026SWE-Duel: Towards a Self-Scaling Arena for Coding Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
  2. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  3. 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation
  4. 2025LLMs Judge Themselves: A Game-Theoretic Framework for Human-Aligned Evaluation
  5. 2024Automated test generation to evaluate tool-augmented LLMs as conversational AI agents
  6. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation
  7. 2025Mitigating Judgment Preference Bias in Large Language Models through Group-Based Polling
  8. 2025Investigating Non-Transitivity in LLM-as-a-Judge