mlsanityData to 7 Oct 2026

How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments

arXiv 2403.11807 ↗First posted 18 Mar 2024cs.AI cs.CLcited by 66

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Decision-making is a complex process requiring diverse abilities, making it an excellent framework for evaluating Large Language Models (LLMs). Researchers have examined LLMs' decision-making through the lens of Game Theory. However, existing evaluation mainly focus on two-player scenarios where an LLM competes against another. Additionally, previous benchmarks suffer from test set leakage due to their static design. We introduce GAMA(γ)-Bench, a new framework for evaluating LLMs' Gaming Ability in Multi-Agent environments. It includes eight classical game theory scenarios and a dynamic scoring scheme specially designed to quantitatively assess LLMs' performance. γ-Bench allows flexible game settings and adapts the scoring system to different game parameters, enabling comprehensive evaluation of robustness, generalizability, and strategies for improvement. Our results indicate that GPT-3.5 demonstrates strong robustness but limited generalizability, which can be enhanced using methods like Chain-of-Thought. We also evaluate 13 LLMs from 6 model families, including GPT-3.5, GPT-4, Gemini, LLaMA-3.1, Mixtral, and Qwen-2. Gemini-1.5-Pro outperforms others, scoring of 69.8 out of 100, followed by LLaMA-3.1-70B (65.9) and Mixtral-8x22B (62.4). Our code and experimental results are publicly available at https://github.com/CUHK-ARISE/GAMABench.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 86 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

36 of 86 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaboration14 Nov
  2. 2023Leveraging Word Guessing Games to Assess the Intelligence of Large Language Models31 Oct
  3. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game29 Oct
  4. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agents2 Oct
  5. 2023Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-429 Sep

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2024Mixtral of Expertscited by 1,785
  6. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  7. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  8. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  9. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  10. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  11. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  12. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
19 more
  1. 2023Better Zero-Shot Reasoning with Role-Play Promptingcited by 220
  2. 2023Evaluating Large Language Models in Theory of Mind Taskscited by 217
  3. 2023Playing repeated games with Large Language Modelscited by 213
  4. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  5. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  6. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 102
  7. 2023Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiationcited by 94
  8. 2023Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysiscited by 91
  9. 2023Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arenacited by 78
  10. 2023ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmarkcited by 77
  11. 2023LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplaycited by 62
  12. 2024Can Large Language Model Agents Simulate Human Trust Behavior?cited by 62
  13. 2023Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Modelscited by 46
  14. 2023GameEval: Evaluating LLMs on Conversational Gamescited by 35
  15. 2023Can Large Language Models Play Text Games Well? Current State-of-the-Art and Open Questionscited by 30
  16. 2023Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluationcited by 24
  17. 2023ChatGPT and Other Large Language Models as Evolutionary Engines for Online Interactive Collaborative Game Designcited by 20
  18. 2023Long-Horizon Dialogue Understanding for Role Identification in the Game of Avalon with Large Language Modelscited by 12
  19. 2023Assessing Large Language Models' ability to predict how humans balance self-interest and the interest of otherscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 28; 2026: 16.

66 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2024OASIS: Open Agent Social Interaction Simulations with One Million Agentscited by 127
  4. 2025Agentic Large Language Models, a surveycited by 105
  5. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  6. 2024Embodied LLM Agents Learn to Cooperate in Organized Teamscited by 81
  7. 2024LLM-based Multi-Agent Reinforcement Learning: Current and Future Directionscited by 67
  8. 2024Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thoughtcited by 56
  9. 2024Game-theoretic LLM: Agent Workflow for Negotiation Gamescited by 55
  10. 2025PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathologycited by 36
  11. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 30
  12. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
38 more
  1. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  2. 2024STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Makingcited by 22
  3. 2025DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environmentscited by 22
  4. 2024SelfGoal: Your Language Agents Already Know How to Achieve High-level Goalscited by 21
  5. 2024TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMscited by 21
  6. 2024How Well Can LLMs Echo Us? Evaluating AI Chatbots' Role-Play Ability with ECHOcited by 19
  7. 2024GLEE: A Unified Framework and Benchmark for Language-based Economic Environmentscited by 19
  8. 2024Toward LLM-Agent-Based Modeling of Transportation Systems: A Conceptual Frameworkcited by 19
  9. 2025Are Large Vision Language Models Good Game Players?cited by 19
  10. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Gamescited by 16
  11. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  12. 2025Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontierscited by 14
  13. 2025LLM Strategic Reasoning: Agentic Study through Behavioral Game Theorycited by 14
  14. 2024A Survey on Human-Centric LLMscited by 13
  15. 2025Self-Resource Allocation in Multi-Agent LLM Systemscited by 13
  16. 2025Shapley-Coop: Credit Assignment for Emergent Cooperation in Self-Interested LLM Agentscited by 11
  17. 2025Distributive Fairness in Large Language Models: Evaluating Alignment with Human Valuescited by 9
  18. 2025A Survey on Enhancing Causal Reasoning Ability of Large Language Modelscited by 8
  19. 2025EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM Agentscited by 8
  20. 2024MALLM-GAN: Multi-Agent Large Language Model as Generative Adversarial Network for Synthesizing Tabular Datacited by 7
  21. 2024Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Modelscited by 7
  22. 2025Towards Strategic Persuasion with Language Modelscited by 7
  23. 2025PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilitiescited by 6
  24. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environmentscited by 5
  25. 2026Think Fast and Slow: Step-Level Cognitive Depth Adaptation for LLM Agentscited by 5
  26. 2026PokerSkill: LLMs Can Play Expert-Level Poker without Training or Solverscited by 5
  27. 2024Integrated Design and Governance of Agentic AI Systems through Adaptive Information Modulationcited by 4
  28. 2025RBF++: Quantifying and Optimizing Reasoning Boundaries across Measurable and Unmeasurable Capabilities for Chain-of-Thought Reasoningcited by 4
  29. 2025Evaluating Language Models' Evaluations of Gamescited by 4
  30. 2024Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Gamecited by 3
  31. 2025Adversarial Testing in LLMs: Insights into Decision-Making Vulnerabilitiescited by 3
  32. 2025TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Gamescited by 3
  33. 2026What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understandingcited by 3
  34. 2024Are Large Language Models Strategic Decision Makers? A Study of Performance and Bias in Two-Player Non-Zero-Sum Gamescited by 2
  35. 2025PoliCon: Evaluating LLMs on Achieving Diverse Political Consensus Objectivescited by 2
  36. 2025Opponent Shaping in LLM Agentscited by 2
  37. 2025LLM-Guided Reinforcement Learning with Representative Agents for Traffic Modelingcited by 2
  38. 2026ALIGN: Aligned Delegation with Performance Guarantees for Multi-Agent LLM Reasoningcited by 2

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMs May Not Be Human-Level Players, But They Can Be Testers: Measuring Game Difficulty with LLM Agents
  2. 2025Meta-Evaluating Local LLMs: Rethinking Performance Metrics for Serious Games
  3. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  4. 2024Do LLM Agents Have Regret? A Case Study in Online Learning and Games
  5. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environments
  6. 2025Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Play
  7. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
  8. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games