mlsanityData to 14 Oct 2026

LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Models

arXiv 2310.03903 ↗First posted 5 Oct 2023cs.CL cs.MAcited by 68

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have demonstrated emergent common-sense reasoning and Theory of Mind (ToM) capabilities, making them promising candidates for developing coordination agents. This study introduces the LLM-Coordination Benchmark, a novel benchmark for analyzing LLMs in the context of Pure Coordination Settings, where agents must cooperate to maximize gains. Our benchmark evaluates LLMs through two distinct tasks. The first is Agentic Coordination, where LLMs act as proactive participants in four pure coordination games. The second is Coordination Question Answering (CoordQA), which tests LLMs on 198 multiple-choice questions across these games to evaluate three key abilities: Environment Comprehension, ToM Reasoning, and Joint Planning. Results from Agentic Coordination experiments reveal that LLM-Agents excel in multi-agent coordination settings where decision-making primarily relies on environmental variables but face challenges in scenarios requiring active consideration of partners' beliefs and intentions. The CoordQA experiments further highlight significant room for improvement in LLMs' Theory of Mind reasoning and joint planning capabilities. Zero-Shot Coordination (ZSC) experiments in the Agentic Coordination setting demonstrate that LLM agents, unlike RL methods, exhibit robustness to unseen partners. These findings indicate the potential of LLMs as Agents in pure coordination setups and underscore areas for improvement. Code Available at https://github.com/eric-ai-lab/llm_coordination.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 30 of its 67 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

30 of 67 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  8. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  9. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  10. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  11. 2017Population Based Training of Neural Networkscited by 367
  12. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
18 more
  1. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  2. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  3. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 241
  4. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  5. 2020"Other-Play" for Zero-Shot Coordinationcited by 198
  6. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 197
  7. 2021Collaborating with Humans without Human Datacited by 142
  8. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 133
  9. 2021Off-Belief Learningcited by 75
  10. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 70
  11. 2018Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 68
  12. 2020A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Taskscited by 52
  13. 2019Simplified Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 48
  14. 2019Two Body Problem: Collaborative Visual Task Completioncited by 45
  15. 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
  16. 2023Cooperative Open-ended Learning Framework for Zero-shot Coordinationcited by 30
  17. 2021Evaluating the Robustness of Collaborative Agentscited by 25
  18. 2023PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordinationcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 28; 2026: 34.

68 papers.

Sort:

By area:

  1. 2025Why Do Multi-Agent LLM Systems Fail?cited by 604
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  4. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  5. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 99
  6. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  7. 2025AgentsNet: Coordination and Collaborative Reasoning in Multi-Agent LLMscited by 31
  8. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  9. 2024Why Solving Multi-agent Path Finding with Large Language Model has not Succeeded Yetcited by 22
  10. 2025Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoningcited by 21
  11. 2026Large Language Model Reasoning Failurescited by 21
  12. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
38 more
  1. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 16
  2. 2025Self-Resource Allocation in Multi-Agent LLM Systemscited by 15
  3. 2024Enhancing Multi-Robot Semantic Navigation Through Multimodal Chain-of-Thought Score Collaborationcited by 14
  4. 2025The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societiescited by 14
  5. 2025Accelerating Earth Science Discovery via Multi-Agent LLM Systemscited by 13
  6. 2025The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperationcited by 13
  7. 2025Benchmarking LLMs' Swarm intelligencecited by 10
  8. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMscited by 10
  9. 2025AgentArch: A Comprehensive Benchmark to Evaluate Agent Architectures in Enterprisecited by 8
  10. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environmentscited by 6
  11. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  12. 2025AlphaApollo: A System for Deep Agentic Reasoningcited by 5
  13. 2026Tacit Coordination of Large Language Modelscited by 5
  14. 2026Tool Use Enables Undetectable Steganography in Multi-Agent LLM Systemscited by 5
  15. 2026Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitorscited by 5
  16. 2024Integrated Design and Governance of Agentic AI Systems through Adaptive Information Modulationcited by 4
  17. 2025PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environmentscited by 4
  18. 2025DixitWorld: Evaluating Multimodal Abductive Reasoning in Vision-Language Models with Multi-Agent Dixit Gameplaycited by 4
  19. 2026Agentic Design Patterns: A System-Theoretic Frameworkcited by 4
  20. 2025When Disagreements Elicit Robustness: Investigating Self-Repair Capabilities under LLM Multi-Agent Disagreementscited by 3
  21. 2025ELHPlan: Efficient Long-Horizon Task Planning for Multi-Agent Collaborationcited by 3
  22. 2026Game-Theoretic Lens on LLM-based Multi-Agent Systemscited by 3
  23. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 3
  24. 2025Learning Robust Social Strategies with Large Language Modelscited by 2
  25. 2026Enhancing Model Context Protocol (MCP) with Context-Aware Server Collaborationcited by 2
  26. 2026DPBench: Structural Determinants of Multi-Agent LLM Coordination Under Simultaneous Resource Contentioncited by 2
  27. 2026Coordination as an Architectural Layer for LLM-Based Multi-Agent Systemscited by 2
  28. 2026LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learningcited by 2
  29. 2026Benchmarking Open-Ended Multi-Agent Coordination in Language Agentscited by 2
  30. 2025Algorithmic Prompt Generation for Diverse Human-like Teaming and Communication with Large Language Modelscited by 1
  31. 2025Is Your LLM Really Mastering the Concept? A Multi-Agent Benchmarkcited by 1
  32. 2025TeamMedAgents: Pareto-Efficient Multi-Agent Medical Reasoning Through Teamwork Theorycited by 1
  33. 2025Decentralized Multi-Agent Goal Assignment for Path Planning using Large Language Modelscited by 1
  34. 2026Advances and Innovations in the Multi-Agent Robotic System (MARS) Challengecited by 1
  35. 2026A Note on the Strategic Confinement Problemcited by 1
  36. 2026Do LLMs Beat Nash? Testing Decentralized Coordination in Self-Play Multi-Agent Gamescited by 1
  37. 2026Testing Interchangeability in LLM Agent Teamscited by 1
  38. 2026CoMemBench: Benchmarking Collaborative Memory Boundaries across Multi-Agent Workflow Topologiescited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Theory of Mind for Multi-Agent Collaboration via Large Language Models
  2. 2026Adaptive Theory of Mind for LLM-based Multi-Agent Coordination
  3. 2024AgentCoord: Visually Exploring Coordination Strategy for LLM-based Multi-Agent Collaboration
  4. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents
  5. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments
  6. 2025On the Importance of Task Complexity in Evaluating LLM-Based Multi-Agent Systems
  7. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents
  8. 2025Towards Cognitive Synergy in LLM-Based Multi-Agent Systems: Integrating Theory of Mind and Critical Evaluation