mlsanityData to 7 Oct 2026

MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration

arXiv 2311.08562 ↗First posted 14 Nov 2023cs.CLcited by 56

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Large Language Models (LLMs) have significantly advanced natural language processing, demonstrating exceptional reasoning, tool usage, and memory capabilities. As their applications expand into multi-agent environments, there arises a need for a comprehensive evaluation framework that captures LLMs' reasoning, planning, collaboration, and other social abilities. This work introduces a novel competition-based benchmark framework specifically designed to assess LLMs within multi-agent settings, providing quantitative metrics to evaluate their judgment, reasoning, deception, self-awareness, cooperation, coordination, and rationality. We utilize two social deduction games alongside three game-theory scenarios to create diverse environments. Our frame is fortified with the probabilistic graphic modeling (PGM) method, enhancing the LLMs' capabilities in navigating complex social and cognitive dimensions. We evaluate seven LLMs, quantitatively highlighting a significant capability gap of over threefold between the strongest, GPT o1, and the weakest, Llama-2-70B. It also confirms that our PGM enhancement boosts the abilities of all selected models by an average of 37%. Our data and code can be found here https://github.com/cathyxl/MAgIC.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 46 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

16 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agents2 Oct
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 13

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  9. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  10. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  11. 2023Playing repeated games with Large Language Modelscited by 213
  12. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
1 more
  1. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 31; 2026: 14.

56 papers.

Sort:

By area:

  1. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  2. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  3. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  4. 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agentscited by 169
  5. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  6. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  7. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  8. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  9. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 89
  10. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 66
  11. 2024Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?cited by 62
  12. 2024A Survey on LLM-based Multi-Agent System: Recent Advances and New Frontiers in Applicationcited by 54
38 more
  1. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  2. 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agentscited by 31
  3. 2025AgentsNet: Coordination and Collaborative Reasoning in Multi-Agent LLMscited by 31
  4. 2025PiFlow: Principle-Aware Scientific Discovery with Multi-Agent Collaborationcited by 25
  5. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  6. 2025DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environmentscited by 22
  7. 2024TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMscited by 21
  8. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  9. 2025Augmenting Multi-Agent Communication with State Delta Trajectorycited by 13
  10. 2025Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Datacited by 12
  11. 2025BannerAgency: Advertising Banner Design with Multimodal LLM Agentscited by 11
  12. 2025Shapley-Coop: Credit Assignment for Emergent Cooperation in Self-Interested LLM Agentscited by 11
  13. 2025Exploring Advanced LLM Multi-Agent Systems Based on Blackboard Architecturecited by 10
  14. 2025Bayesian Social Deduction with Graph-Informed Language Modelscited by 8
  15. 2025CausalMACE: Causality Empowered Multi-Agents in Minecraft Cooperative Taskscited by 8
  16. 2025Long Context Scaling: Divide and Conquer via Multi-Agent Question-driven Collaborationcited by 7
  17. 2025Towards Strategic Persuasion with Language Modelscited by 7
  18. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignmentcited by 7
  19. 2025CoMet: Metaphor-Driven Covert Communication for Multi-Agent Language Gamescited by 6
  20. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environmentscited by 5
  21. 2025AI Agent Behavioral Sciencecited by 5
  22. 2024Integrated Design and Governance of Agentic AI Systems through Adaptive Information Modulationcited by 4
  23. 2025Communication Enables Cooperation in LLM Agents: A Comparison with Curriculum-Based Approachescited by 4
  24. 2025How Far Can LLMs Improve from Experience? Measuring Test-Time Learning Ability in LLMs with Human Comparisoncited by 3
  25. 2026Agentic AI Meets Edge Computing in Autonomous UAV Swarmscited by 3
  26. 2026Byzantine Cheap Talk: Adversarial Resilience and Topology Effects in LLM Coordination Gamescited by 3
  27. 2025LLMs for Supply Chain Managementcited by 2
  28. 2026From Prompts to Pavement: LMMs-based Agentic Behavior-Tree Generation Framework for Autonomous Vehiclescited by 2
  29. 2026CollabSim: A CSCW-Grounded Methodology for Investigating Collaborative Competence of LLM Agents through Controlled Multi-Agent Experimentscited by 2
  30. 2026When "Must" Becomes "Maybe": Constraint Weakening in LLM Agent Workflowscited by 2
  31. 2025An agentic system with reinforcement-learned subsystem improvements for parsing form-like documentscited by 1
  32. 2025Is Your LLM Really Mastering the Concept? A Multi-Agent Benchmarkcited by 1
  33. 2025WGSR-Bench: Wargame-based Game-theoretic Strategic Reasoning Benchmark for Large Language Modelscited by 1
  34. 2025FAIRGAMER: Evaluating Social Biases in LLM-Based Video Game NPCscited by 1
  35. 2025Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Reviewcited by 0
  36. 2025COCORELI: Enforcing Execution Preconditions for Reliable Collaborative Instruction Followingcited by 0
  37. 2026When Identity Overrides Incentives: Representational Choices as Governance Decisions in Multi-Agent LLM Systemscited by 0
  38. 2026Beneath the Surface: Investigating LLMs' Capabilities for Communicating with Subtextcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments
  2. 2025Grounding Natural Language for Multi-agent Decision-Making with Multi-agentic LLMs
  3. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  4. 2026MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs
  5. 2024A Survey on Large Language Model-Based Game Agents
  6. 2025Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia
  7. 2024Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash
  8. 2025Playing games with Large language models: Randomness and strategy