mlsanityData to 7 Oct 2026

MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents

arXiv 2503.01935 ↗First posted 3 Mar 2025cs.MA cs.AI cs.CLcited by 169

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable capabilities as autonomous agents, yet existing benchmarks either focus on single-agent tasks or are confined to narrow domains, failing to capture the dynamics of multi-agent coordination and competition. In this paper, we introduce MultiAgentBench, a comprehensive benchmark designed to evaluate LLM-based multi-agent systems across diverse, interactive scenarios. Our framework measures not only task completion but also the quality of collaboration and competition using novel, milestone-based key performance indicators. Moreover, we evaluate various coordination protocols (including star, chain, tree, and graph topologies) and innovative strategies such as group discussion and cognitive planning. Notably, gpt-4o-mini reaches the average highest task score, graph structure performs the best among coordination protocols in the research scenario, and cognitive planning improves milestone achievement rates by 3%. Code and datasets are public available at https://github.com/MultiagentBench/MARBLE.

Key facts. Cited by 169 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 24 of its 52 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

24 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solving18 May

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  8. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  9. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  10. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  11. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
  12. 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repaircited by 225
11 more
  1. 2023War and Peace (WarAgent): Large Language Model-based Multi-Agent Simulation of World Warscited by 143
  2. 2023Agents: An Open-source Framework for Autonomous Language Agentscited by 116
  3. 2024Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimizationcited by 84
  4. 2024LitLLM: A Toolkit for Scientific Literature Reviewcited by 61
  5. 2023MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaborationcited by 56
  6. 2023GameGPT: Multi-agent Collaborative Framework for Game Developmentcited by 41
  7. 2024Large Language Model for Participatory Urban Planningcited by 40
  8. 2024VillagerAgent: A Graph-Based Multi-Agent Framework for Coordinating Complex Task Dependencies in Minecraftcited by 34
  9. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  10. 2024Enhancing Diagnostic Accuracy through Multi-Agent Conversations: Using Large Language Models to Mitigate Cognitive Biascited by 20
  11. 2024CREW: Facilitating Human-AI Teaming Researchcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 49; 2026: 120.

169 papers.

Sort:

By area:

  1. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  3. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  4. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2025Where LLM Agents Fail and How They can Learn From Failurescited by 118
  7. 2025UserBench: An Interactive Gym Environment for User-Centric Agentscited by 69
  8. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  9. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  10. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  11. 2025Who is Introducing the Failure? Automatically Attributing Failures of Multi-Agent Systems via Spectrum Analysiscited by 33
  12. 2025Enabling Agents to Communicate Entirely in Latent Spacecited by 31
38 more
  1. 2025ARE: Scaling Up Agent Environments and Evaluationscited by 28
  2. 2025UserRL: Training Interactive User-Centric Agent via Reinforcement Learningcited by 27
  3. 2025Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attributioncited by 25
  4. 2025AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systemscited by 23
  5. 2025Creativity in LLM-based Multi-Agent Systems: A Surveycited by 23
  6. 2025Attack the Messages, Not the Agents: A Multi-round Adaptive Stealthy Tampering Framework for LLM-MAScited by 17
  7. 2025Risk Analysis Techniques for Governed LLM-based Multi-Agent Systemscited by 16
  8. 2025Must Read: A Comprehensive Survey of Computational Persuasioncited by 12
  9. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  10. 2026Lightweight LLM Agent Memory with Small Language Modelscited by 12
  11. 2026Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agentscited by 11
  12. 2025From What to Why: A Multi-Agent System for Evidence-based Chemical Reaction Condition Reasoningcited by 10
  13. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMscited by 10
  14. 2025TAMA: A Human-AI Collaborative Thematic Analysis Framework Using Multi-Agent LLMs for Clinical Interviewscited by 9
  15. 2025A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directionscited by 9
  16. 2025Auto-TA: Towards Scalable Automated Thematic Analysis (TA) via Multi-Agent Large Language Models with Reinforcement Learningcited by 9
  17. 2025Verification-Aware Planning for Multi-Agent Systemscited by 9
  18. 2026SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agentscited by 9
  19. 2025LLM Applications: Current Paradigms and the Next Frontiercited by 8
  20. 2025Benchmarking LLMs' Swarm intelligencecited by 8
  21. 2025AgentArch: A Comprehensive Benchmark to Evaluate Agent Architectures in Enterprisecited by 8
  22. 2025Orchestrating Human-AI Teams: The Manager Agent as a Unifying Research Challengecited by 8
  23. 2026M3-BENCH: Process-Aware Evaluation of LLM Agents' Social Behaviors in Mixed-Motive Gamescited by 8
  24. 2026CL-bench Life: Can Language Models Learn from Real-Life Context?cited by 8
  25. 2026TeamBench: Evaluating Agent Coordination under Enforced Role Separationcited by 8
  26. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Criticcited by 7
  27. 2026AgentSocialBench: Evaluating Privacy Risks in Human-Centered Agentic Social Networkscited by 7
  28. 2026AutoScientists: Self-Organizing Agent Teams for Long-Running Scientific Experimentationcited by 7
  29. 2025HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worldscited by 6
  30. 2025Agentic Services Computingcited by 6
  31. 2025AgentAsk: Multi-Agent Systems Need to Askcited by 6
  32. 2026Generative AI collective behavior needs an interactionist paradigmcited by 6
  33. 2026Evaluating Collective Behaviour of Hundreds of LLM Agentscited by 6
  34. 2026CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMscited by 6
  35. 2026LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systemscited by 6
  36. 2025Topological Structure Learning Should Be A Research Priority for LLM-Based Multi-Agent Systemscited by 5
  37. 2025Agent Identity Evals: Measuring Agentic Identitycited by 5
  38. 2025CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressurescited by 5

50 most cited of 169 All 169 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024BattleAgentBench: A Benchmark for Evaluating Cooperation and Competition Capabilities of Language Models in Multi-Agent Systems
  2. 2026MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability
  3. 2026Benchmarking Emergent Coordination in Large-Scale LLM Populations: An Evaluation Framework on the MoltBook Archive
  4. 2025LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systems
  5. 2023AgentBench: Evaluating LLMs as Agents
  6. 2024AgentCoord: Visually Exploring Coordination Strategy for LLM-based Multi-Agent Collaboration
  7. 2025Can LLM Agents Solve Collaborative Tasks? A Study on Urgency-Aware Planning and Coordination
  8. 2025AgentsNet: Coordination and Collaborative Reasoning in Multi-Agent LLMs