mlsanityData to 7 Oct 2026

GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents

arXiv 2406.06613 ↗First posted 7 Jun 2024cs.CL cs.AIcited by 89

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Large language models have demonstrated remarkable few-shot performance on many natural language understanding tasks. Despite several demonstrations of using large language models in complex, strategic scenarios, there lacks a comprehensive framework for evaluating agents' performance across various types of reasoning found in games. To address this gap, we introduce GameBench, a cross-domain benchmark for evaluating strategic reasoning abilities of LLM agents. We focus on 9 different game environments, where each covers at least one axis of key reasoning skill identified in strategy games, and select games for which strategy explanations are unlikely to form a significant portion of models' pretraining corpuses. Our evaluations use GPT-3 and GPT-4 in their base form along with two scaffolding frameworks designed to enhance strategic reasoning ability: Chain-of-Thought (CoT) prompting and Reasoning Via Planning (RAP). Our results show that none of the tested models match human performance, and at worst GPT-4 performs worse than random action. CoT and RAP both improve scores but not comparable to human levels.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 21 of its 43 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

21 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations19 Feb
  2. 2023Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena9 Oct
  3. 2023Strategic Reasoning with Language Models30 May
  4. 2023Reasoning with Language Model is Planning with World Model24 May

Other references · 17

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  4. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  5. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  6. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  7. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  8. 2023Playing repeated games with Large Language Modelscited by 213
  9. 2023SmartPlay: A Benchmark for LLMs as Intelligent Agentscited by 123
  10. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 94
  11. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  12. 2023MAgIC: Investigation of Large Language Model Powered Multi-Agent in Cognition, Adaptability, Rationality and Collaborationcited by 56
5 more
  1. 2023How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environmentcited by 55
  2. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agentscited by 52
  3. 2023GameEval: Evaluating LLMs on Conversational Gamescited by 35
  4. 2024LLMArena: Assessing Capabilities of Large Language Models in Dynamic Multi-Agent Environmentscited by 35
  5. 2023ALYMPICS: LLM Agents Meet Game Theory -- Exploring Strategic Decision-Making with AI Agentscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 39; 2026: 43.

89 papers.

Sort:

By area:

  1. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  2. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  3. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  4. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 30
  5. 2025Code World Models for General Game Playingcited by 25
  6. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  7. 2025TextArenacited by 24
  8. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  9. 2024Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboardcited by 22
  10. 2025DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environmentscited by 22
  11. 2025Measuring General Intelligence with Generated Gamescited by 21
  12. 2025Agents of Change: Self-Evolving LLM Agents for Strategic Planningcited by 20
38 more
  1. 2025Are Large Vision Language Models Good Game Players?cited by 19
  2. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  3. 2024Subversion Strategy Eval: Can language models statelessly strategize to subvert control protocols?cited by 14
  4. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  5. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Modelscited by 10
  6. 2025Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applicationscited by 10
  7. 2025Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Playcited by 10
  8. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 9
  9. 2026WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Gamescited by 9
  10. 2024Codenames as a Benchmark for Large Language Modelscited by 8
  11. 2026M3-BENCH: Process-Aware Evaluation of LLM Agents' Social Behaviors in Mixed-Motive Gamescited by 8
  12. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  13. 2026How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Usecited by 7
  14. 2026SoK: Agentic Retrieval-Augmented Generation (RAG): Taxonomy, Architectures, Evaluation, and Research Directionscited by 7
  15. 2025Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacycited by 6
  16. 2025ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competitioncited by 5
  17. 2025ZeroSumEval: Scaling LLM Evaluation with Inter-Model Competitioncited by 5
  18. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Makingcited by 5
  19. 2026MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMscited by 5
  20. 2024Improving Cooperation in Language Games with Bayesian Inference and the Cognitive Hierarchycited by 4
  21. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  22. 2025ModelingAgent: Bridging LLMs and Mathematical Modeling for Real-World Challengescited by 4
  23. 2025A Practical Approach for Building Production-Grade Conversational Agents with Workflow Graphscited by 4
  24. 2025Communication Enables Cooperation in LLM Agents: A Comparison with Curriculum-Based Approachescited by 4
  25. 2025Vox Deorum: A Hybrid LLM Architecture for 4X / Grand Strategy Game AI -- Lessons from Civilization Vcited by 4
  26. 2025TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Gamescited by 3
  27. 2025A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembenchcited by 3
  28. 2026What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understandingcited by 3
  29. 2026Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargainingcited by 3
  30. 2025Cardiverse: Harnessing LLMs for Novel Card Game Prototypingcited by 2
  31. 2025Agents Require Metacognitive and Strategic Reasoning to Succeed in the Coming Labor Marketscited by 2
  32. 2025Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMscited by 2
  33. 2025UrzaGPT: LoRA-Tuned Large Language Models for Card Selection in Collectible Card Gamescited by 2
  34. 2026Game-Theoretic Lens on LLM-based Multi-Agent Systemscited by 2
  35. 2026BotzoneBench: Scalable LLM Evaluation via Graded AI Anchorscited by 2
  36. 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMscited by 2
  37. 2026CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization Vcited by 2
  38. 2026Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiversecited by 2

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMs
  2. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games
  3. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  4. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games
  5. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models
  6. 2025LLM Strategic Reasoning: Agentic Study through Behavioral Game Theory
  7. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
  8. 2026Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments