# PuzzlePlex: Benchmarking Foundation Models on Reasoning and Planning with Puzzles

arXiv: https://arxiv.org/abs/2510.06475 · HTML: https://mlsanity.com/paper/2510.06475/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 7 Oct 2025.

## Builds on (most cited first, 18 of 18)

- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [SmartPlay: A Benchmark for LLMs as Intelligent Agents](https://mlsanity.com/paper/2310.01557/) (2023)
- [LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models](https://mlsanity.com/paper/2311.18232/) (2023)
- [ChessGPT: Bridging Policy Learning and Language Modeling](https://mlsanity.com/paper/2306.09200/) (2023)
- [Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation](https://mlsanity.com/paper/2311.04254/) (2023)
- [RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge](https://mlsanity.com/paper/2101.00376/) (2021)
- [Elo Uncovered: Robustness and Best Practices in Language Model Evaluation](https://mlsanity.com/paper/2311.17295/) (2023)
- [BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information](https://mlsanity.com/paper/2306.07934/) (2023)
- [Programming Puzzles](https://mlsanity.com/paper/2106.05784/) (2021)
- [EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges](https://mlsanity.com/paper/2502.08859/) (2025)
- [PuzzleBench: A Fully Dynamic Evaluation Framework for Large Multimodal Models on Puzzle Solving](https://mlsanity.com/paper/2504.10885/) (2025)
- [FCoReBench: Can Large Language Models Solve Challenging First-Order Combinatorial Reasoning Problems?](https://mlsanity.com/paper/2402.02611/) (2024)
- [Semantic Strengthening of Neuro-Symbolic Learning](https://mlsanity.com/paper/2302.14207/) (2023)
- [Puzzle Solving without Search or Human Knowledge: An Unnatural Language Approach](https://mlsanity.com/paper/2109.02797/) (2021)
- [Are LLMs Good Cryptic Crossword Solvers?](https://mlsanity.com/paper/2403.12094/) (2024)
- [Composing Neural Learning and Symbolic Reasoning with an Application to Visual Discrimination](https://mlsanity.com/paper/1907.05878/) (2019)
- [Heuristics, Answer Set Programming and Markov Decision Process for Solving a Set of Spatial Puzzles](https://mlsanity.com/paper/1903.03411/) (2019)

## Built on by (most cited first, 5 of 5)

- [TopoBench: Benchmarking LLMs on Hard Topological Reasoning](https://mlsanity.com/paper/2603.12133/) (2026)
- [Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse](https://mlsanity.com/paper/2605.09965/) (2026)
- [Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzles](https://mlsanity.com/paper/2601.13705/) (2026)
- [Failing to Explore: Language Models on Interactive Tasks](https://mlsanity.com/paper/2601.22345/) (2026)
- [The Token Games: Evaluating Language Model Reasoning with Puzzle Duels](https://mlsanity.com/paper/2602.17831/) (2026)

## Abstract (arXiv)

This work investigates the reasoning and planning capabilities of foundation models and their scalability in complex, dynamic environments. We introduce PuzzlePlex, a benchmark designed to assess these capabilities through a diverse set of puzzles. PuzzlePlex consists of 15 types of puzzles, including deterministic and stochastic games of varying difficulty, as well as single-player and two-player scenarios. The PuzzlePlex framework provides a comprehensive environment for each game, and supports extensibility to generate more challenging instances as foundation models evolve. Additionally, we implement customized game-playing strategies for comparison. Building on this benchmark, we develop fine-grained metrics to measure performance and conduct an in-depth analysis of frontier foundation models across two settings: instruction-based and code-based. Furthermore, we systematically investigate their scaling limits. Our findings show that reasoning models outperform others in instruction-based settings, while code-based execution presents greater challenges but offers a scalable and efficient alternative. PuzzlePlex enables targeted evaluation and guides future improvements in reasoning, planning, and generalization for foundation models.