# Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing

arXiv: https://arxiv.org/abs/2602.19160 · HTML: https://mlsanity.com/paper/2602.19160/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 22 Feb 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 25 of 25)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning](https://mlsanity.com/paper/2205.09712/) (2022)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [Constructing Datasets for Multi-hop Reading Comprehension Across Documents](https://mlsanity.com/paper/1710.06481/) (2017)
- [Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis](https://mlsanity.com/paper/2312.05488/) (2023)
- [LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2404.15522/) (2024)
- [Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyond](https://mlsanity.com/paper/2306.09841/) (2023)
- [Logical Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2502.09100/) (2025)
- [lmgame-Bench: How Good are LLMs at Playing Games?](https://mlsanity.com/paper/2505.15146/) (2025)
- [Mastering Board Games by External and Internal Planning with Language Models](https://mlsanity.com/paper/2412.12119/) (2024)
- [History, Development, and Principles of Large Language Models-An Introductory Survey](https://mlsanity.com/paper/2402.06853/) (2024)
- [Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs](https://mlsanity.com/paper/2402.11442/) (2024)
- [Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboard](https://mlsanity.com/paper/2407.07796/) (2024)
- [To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models](https://mlsanity.com/paper/2412.15584/) (2024)
- [Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models](https://mlsanity.com/paper/2410.02205/) (2024)
- [Revisiting the Othello World Model Hypothesis](https://mlsanity.com/paper/2503.04421/) (2025)
- [GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games](https://mlsanity.com/paper/2508.08501/) (2025)
- [Can Large Language Models Act as Symbolic Reasoners?](https://mlsanity.com/paper/2410.21490/) (2024)
- [Grammar-based Game Description Generation using Large Language Models](https://mlsanity.com/paper/2407.17404/) (2024)

## Abstract (arXiv)

This paper examines the reasoning capabilities of Large Language Models (LLMs) from a novel perspective, focusing on their ability to operate within formally specified, rule-governed environments. We evaluate four LLMs (Gemini 2.5 Pro and Flash variants, Llama 3.3 70B and GPT-OSS 120B) on a suite of forward-simulation tasks-including next / multistep state formulation, and legal action generation-across a diverse set of reasoning problems illustrated through General Game Playing (GGP) game instances. Beyond reporting instance-level performance, we characterize games based on 40 structural features and analyze correlations between these features and LLM performance. Furthermore, we investigate the effects of various game obfuscations to assess the role of linguistic semantics in game definitions and the impact of potential prior exposure of LLMs to specific games during training. The main results indicate that three of the evaluated models generally perform well across most experimental settings, with performance degradation observed as the evaluation horizon increases (i.e., with a higher number of game steps). Detailed case-based analysis of the LLM performance provides novel insights into common reasoning errors in the considered logic-based problem formulation, including hallucinated rules, redundant state facts, or syntactic errors. Overall, the paper reports clear progress in formal reasoning capabilities of contemporary models.