mlsanityData to 7 Oct 2026

GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations

arXiv 2402.12348 ↗First posted 19 Feb 2024cs.CL cs.AI cs.LGcited by 118

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

As Large Language Models (LLMs) are integrated into critical real-world applications, their strategic and logical reasoning abilities are increasingly crucial. This paper evaluates LLMs' reasoning abilities in competitive environments through game-theoretic tasks, e.g., board and card games that require pure logic and strategic reasoning to compete with opponents. We first propose GTBench, a language-driven environment composing 10 widely recognized tasks, across a comprehensive game taxonomy: complete versus incomplete information, dynamic versus static, and probabilistic versus deterministic scenarios. Then, we (1) Characterize the game-theoretic reasoning of LLMs; and (2) Perform LLM-vs.-LLM competitions as reasoning evaluation. We observe that (1) LLMs have distinct behaviors regarding various gaming scenarios; for example, LLMs fail in complete and deterministic games yet they are competitive in probabilistic gaming scenarios; (2) Most open-source LLMs, e.g., CodeLlama-34b-Instruct and Llama-2-70b-chat, are less competitive than commercial LLMs, e.g., GPT-4, in complex games, yet the recently released Llama-3-70b-Instruct makes up for this shortcoming. In addition, code-pretraining greatly benefits strategic reasoning, while advanced reasoning methods such as Chain-of-Thought (CoT) and Tree-of-Thought (ToT) do not always help. We further characterize the game-theoretic properties of LLMs, such as equilibrium and Pareto Efficiency in repeated games. Detailed error profiles are provided for a better understanding of LLMs' behavior. We hope our research provides standardized protocols and serves as a foundation to spur further explorations in the strategic reasoning of LLMs.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 41 of its 64 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

41 of 64 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs22 Sep
  2. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2023Mistral 7Bcited by 3,065
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  11. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  12. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
27 more
  1. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  2. 2023BloombergGPT: A Large Language Model for Financecited by 923
  3. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  4. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  5. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  7. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  8. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  9. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  10. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  11. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  12. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  13. 2023Playing repeated games with Large Language Modelscited by 213
  14. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 180
  15. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  16. 2019Interactive Fiction Games: A Colossal Adventurecited by 165
  17. 2022Language Models of Code are Few-Shot Commonsense Learnerscited by 160
  18. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 148
  19. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  20. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  21. 2022SecureBERT: A Domain-Specific Language Model for Cybersecuritycited by 60
  22. 2023Clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agentscited by 52
  23. 2023Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Modelscited by 46
  24. 2023Welfare Diplomacy: Benchmarking Language Model Cooperationcited by 43
  25. 2023FIREBALL: A Dataset of Dungeons and Dragons Actual-Play with Structured Game State Informationcited by 18
  26. 2023Long-Horizon Dialogue Understanding for Role Identification in the Game of Avalon with Large Language Modelscited by 12
  27. 2016An Experimental Evaluation of Regret-Based Econometricscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 49; 2026: 40.

118 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 120
  3. 2024Emergent social conventions and collective bias in LLM populationscited by 115
  4. 2025Agentic Large Language Models, a surveycited by 105
  5. 2024LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Modelscited by 97
  6. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agentscited by 89
  7. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  8. 2024How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environmentscited by 66
  9. 2026AutoHarness: improving LLM agents by automatically synthesizing a code harnesscited by 63
  10. 2024Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?cited by 62
  11. 2024ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guaranteescited by 62
  12. 2024Game-theoretic LLM: Agent Workflow for Negotiation Gamescited by 55
38 more
  1. 2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 54
  2. 2024Moral Alignment for LLM Agentscited by 39
  3. 2025Parameters vs. Context: Fine-Grained Control of Knowledge Reliance in Language Modelscited by 38
  4. 2025UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Makingcited by 36
  5. 2025G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learningcited by 29
  6. 2025Estimating LLM Uncertainty with Evidencecited by 26
  7. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  8. 2025TextArenacited by 24
  9. 2024A Survey on Large Language Model-Based Social Agents in Game-Theoretic Scenarioscited by 23
  10. 2024Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboardcited by 22
  11. 2025DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environmentscited by 22
  12. 2024LLMs with Personalities in Multi-issue Negotiation Gamescited by 21
  13. 2024TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMscited by 21
  14. 2025CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thoughtcited by 20
  15. 2024GLEE: A Unified Framework and Benchmark for Language-based Economic Environmentscited by 19
  16. 2024Toward LLM-Agent-Based Modeling of Transportation Systems: A Conceptual Frameworkcited by 19
  17. 2025Are Large Vision Language Models Good Game Players?cited by 19
  18. 2024Word-Sequence Entropy: Towards Uncertainty Estimation in Free-Form Medical Question Answering Applications and Beyondcited by 18
  19. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 17
  20. 2025Group size effects and collective misalignment in LLM multi-agent systemscited by 16
  21. 2024Self-Cognition in Large Language Models: An Exploratory Studycited by 14
  22. 2025LLM Strategic Reasoning: Agentic Study through Behavioral Game Theorycited by 14
  23. 2025Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Surveycited by 12
  24. 2025Shapley-Coop: Credit Assignment for Emergent Cooperation in Self-Interested LLM Agentscited by 11
  25. 2025V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Modelscited by 10
  26. 2025Humanizing LLMs: A Survey of Psychological Measurements with Tools, Datasets, and Human-Agent Applicationscited by 10
  27. 2025Grammars of Formal Uncertainty: When to Trust LLMs in Automated Reasoning Taskscited by 9
  28. 2024clembench-2024: A Challenging, Dynamic, Complementary, Multilingual Benchmark and Underlying Flexible Framework for LLMs as Multi-Action Agentscited by 8
  29. 2025EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM Agentscited by 8
  30. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  31. 2024When Reasoning Meets Information Aggregation: A Case Study with Sports Narrativescited by 7
  32. 2024Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Modelscited by 7
  33. 2024SimulBench: Evaluating Language Models with Creative Simulation Taskscited by 7
  34. 2025Approximating Human Strategic Reasoning with LLM-Enhanced Recursive Reasoners Leveraging Multi-agent Hypergamescited by 7
  35. 2025DipSVD: Dual-importance Protected SVD for Efficient LLM Compressioncited by 7
  36. 2025PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilitiescited by 6
  37. 2025Democratizing Diplomacy: A Harness for Evaluating Any Large Language Model on Full-Press Diplomacycited by 6
  38. 2026Neuro-Symbolic Synergy for World Modelingcited by 6

50 most cited of 118 All 118 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing
  2. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games
  3. 2025Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Play
  4. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
  5. 2025CHBench: A Cognitive Hierarchy Benchmark for Evaluating Strategic Reasoning Capability of LLMs
  6. 2026How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Use
  7. 2025Rule Synergy Analysis using LLMs: State of the Art and Implications
  8. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games