mlsanityData to 7 Oct 2026

Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing

arXiv 2602.19160 ↗First posted 22 Feb 2026cs.AI cs.CL cs.LOcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

This paper examines the reasoning capabilities of Large Language Models (LLMs) from a novel perspective, focusing on their ability to operate within formally specified, rule-governed environments. We evaluate four LLMs (Gemini 2.5 Pro and Flash variants, Llama 3.3 70B and GPT-OSS 120B) on a suite of forward-simulation tasks-including next / multistep state formulation, and legal action generation-across a diverse set of reasoning problems illustrated through General Game Playing (GGP) game instances. Beyond reporting instance-level performance, we characterize games based on 40 structural features and analyze correlations between these features and LLM performance. Furthermore, we investigate the effects of various game obfuscations to assess the role of linguistic semantics in game definitions and the impact of potential prior exposure of LLMs to specific games during training. The main results indicate that three of the evaluated models generally perform well across most experimental settings, with performance degradation observed as the evaluation horizon increases (i.e., with a higher number of game steps). Detailed case-based analysis of the LLM performance provides novel insights into common reasoning errors in the considered logic-based problem formulation, including hallucinated rules, redundant state facts, or syntactic errors. Overall, the paper reports clear progress in formal reasoning capabilities of contemporary models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 25 of its 31 references that are in the map. First posted on arXiv 22 Feb 2026.

Builds on

25 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Mastering Board Games by External and Internal Planning with Language Models2 Dec

Other references · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  5. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  6. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  7. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  8. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  9. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  10. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  11. 2023Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysiscited by 91
  12. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 67
12 more
  1. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  2. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  3. 2025Logical Reasoning in Large Language Models: A Surveycited by 52
  4. 2024History, Development, and Principles of Large Language Models-An Introductory Surveycited by 30
  5. 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMscited by 28
  6. 2024Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboardcited by 22
  7. 2024To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Modelscited by 17
  8. 2024Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Modelscited by 12
  9. 2025Revisiting the Othello World Model Hypothesiscited by 11
  10. 2024Can Large Language Models Act as Symbolic Reasoners?cited by 8
  11. 2025GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Gamescited by 8
  12. 2024Grammar-based Game Description Generation using Large Language Modelscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations
  2. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models
  3. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Games
  4. 2024GameArena: Evaluating LLM Reasoning through Live Computer Games
  5. 2025Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making
  6. 2025Game Reasoning Arena: A Framework and Benchmark for Assessing Reasoning Capabilities of Large Language Models via Game Play
  7. 2025Baba is LLM: Reasoning in a Game with Dynamic Rules
  8. 2024GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents