# Explaining Puzzle Solutions in Natural Language: An Exploratory Study on 6x6 Sudoku

arXiv: https://arxiv.org/abs/2505.15993 · HTML: https://mlsanity.com/paper/2505.15993/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 21 May 2025.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Explaining Explanations: An Overview of Interpretability of Machine Learning](https://mlsanity.com/paper/1806.00069/) (2018)
- [Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs](https://mlsanity.com/paper/2210.12283/) (2022)
- [DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data](https://mlsanity.com/paper/2405.14333/) (2024)
- [PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns](https://mlsanity.com/paper/2403.13315/) (2024)
- [Puzzle Solving using Reasoning of Large Language Models: A Survey](https://mlsanity.com/paper/2402.11291/) (2024)
- [Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study](https://mlsanity.com/paper/2311.07387/) (2023)

## Built on by (most cited first, 5 of 5)

- [LLM-Guided Reinforcement Learning with Representative Agents for Traffic Modeling](https://mlsanity.com/paper/2511.06260/) (2025)
- [Toward a Universal Characterization of Sudoku Puzzle Difficulty](https://mlsanity.com/paper/2507.21137/) (2025)
- [PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data](https://mlsanity.com/paper/2508.15180/) (2025)
- [Artificial Effort](https://mlsanity.com/paper/2605.23920/) (2026)
- [Position: Certified Correctness in Neural Constraint Reasoning Requires Symbolic Integration](https://mlsanity.com/paper/2608.14569/) (2026)

## Abstract (arXiv)

The success of Large Language Models (LLMs) in human-AI collaborative decision-making hinges on their ability to provide trustworthy, gradual, and tailored explanations. Solving complex puzzles, such as Sudoku, offers a canonical example of this collaboration, where clear and customized explanations often hold greater importance than the final solution. In this study, we evaluate the performance of five LLMs in solving and explaining sixsix Sudoku puzzles. While one LLM demonstrates limited success in solving puzzles, none can explain the solution process in a manner that reflects strategic reasoning or intuitive problem-solving. These findings underscore significant challenges that must be addressed before LLMs can become effective partners in human-AI collaborative decision-making.