# Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter?

arXiv: https://arxiv.org/abs/2407.14790 · HTML: https://mlsanity.com/paper/2407.14790/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 20 Jul 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 15 of 15)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [Self-Discover: Large Language Models Self-Compose Reasoning Structures](https://mlsanity.com/paper/2402.03620/) (2024)
- [Small Language Models Need Strong Verifiers to Self-Correct Reasoning](https://mlsanity.com/paper/2404.17140/) (2024)
- [LLMs cannot find reasoning errors, but can correct them given the error location](https://mlsanity.com/paper/2311.08516/) (2023)
- [Puzzle Solving using Reasoning of Large Language Models: A Survey](https://mlsanity.com/paper/2402.11291/) (2024)
- [LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles](https://mlsanity.com/paper/2308.10855/) (2023)
- [Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2406.17169/) (2024)
- [Programming Puzzles](https://mlsanity.com/paper/2106.05784/) (2021)
- [Are ChatGPT and GPT-4 Good Poker Players? -- A Pre-Flop Analysis](https://mlsanity.com/paper/2308.12466/) (2023)
- [Solving and Generating NPR Sunday Puzzles with Large Language Models](https://mlsanity.com/paper/2306.12255/) (2023)
- [Piecing Together Clues: A Benchmark for Evaluating the Detective Skills of Large Language Models](https://mlsanity.com/paper/2307.05113/) (2023)

## Built on by (most cited first, 18 of 18)

- [ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning](https://mlsanity.com/paper/2502.01100/) (2025)
- [Evaluating Step-by-step Reasoning Traces: A Survey](https://mlsanity.com/paper/2502.12289/) (2025)
- [SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulas](https://mlsanity.com/paper/2505.14615/) (2025)
- [EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges](https://mlsanity.com/paper/2502.08859/) (2025)
- [TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models](https://mlsanity.com/paper/2506.01341/) (2025)
- [Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning](https://mlsanity.com/paper/2502.05675/) (2025)
- [CP-Agent: Agentic Constraint Programming](https://mlsanity.com/paper/2508.07468/) (2025)
- [Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL](https://mlsanity.com/paper/2509.06024/) (2025)
- [Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoning](https://mlsanity.com/paper/2603.02119/) (2026)
- [A Solver-in-the-Loop Framework for Improving LLMs on Answer Set Programming for Logic Puzzle Solving](https://mlsanity.com/paper/2512.17093/) (2025)
- [TopoBench: Benchmarking LLMs on Hard Topological Reasoning](https://mlsanity.com/paper/2603.12133/) (2026)
- [seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs](https://mlsanity.com/paper/2509.16866/) (2025)
- [The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs using Indian Riddles](https://mlsanity.com/paper/2511.00960/) (2025)
- [Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzles](https://mlsanity.com/paper/2511.06160/) (2025)
- [LLMs for Game Theory: Entropy-Guided In-Context Learning and Adaptive CoT Reasoning](https://mlsanity.com/paper/2601.10775/) (2026)
- [PHANTOM RECALL: When Familiar Puzzles Fool Smart Models](https://mlsanity.com/paper/2510.11812/) (2025)
- [Cognitive bias in LLM reasoning compromises interpretation of clinical oncology notes](https://mlsanity.com/paper/2511.20680/) (2025)
- [AsymPuzl: An Asymmetric Puzzle for multi-agent cooperation](https://mlsanity.com/paper/2512.03466/) (2025)

## Abstract (arXiv)

Solving grid puzzles involves a significant amount of logical reasoning. Hence, it is a good domain to evaluate the reasoning capability of a model which can then guide us to improve the reasoning ability of models. However, most existing works evaluate only the final predicted answer of a puzzle, without delving into an in-depth analysis of the LLMs' reasoning chains (such as where they falter) or providing any finer metrics to evaluate them. Since LLMs may rely on simple heuristics or artifacts to predict the final answer, it is crucial to evaluate the generated reasoning chain beyond overall correctness measures, for accurately evaluating the reasoning abilities of LLMs. To this end, we first develop GridPuzzle, an evaluation dataset comprising 274 grid-based puzzles with different complexities. Second, we propose a new error taxonomy derived from manual analysis of reasoning chains from LLMs including GPT-4, Claude-3, Gemini, Mistral, and Llama-2. Then, we develop an LLM-based framework for large-scale subjective evaluation (i.e., identifying errors) and an objective metric, PuzzleEval, to evaluate the correctness of reasoning chains. Evaluating reasoning chains from LLMs leads to several interesting findings. We further show that existing prompting methods used for enhancing models' reasoning abilities do not improve performance on GridPuzzle. This highlights the importance of understanding fine-grained errors and presents a challenge for future research to enhance LLMs' puzzle-solving abilities by developing methods that address these errors. Data and source code are available at https://github.com/Mihir3009/GridPuzzle.