mlsanityData to 14 Oct 2026

Sudoku-Bench: Evaluating creative reasoning with Sudoku variants

arXiv 2505.16135 ↗First posted 22 May 2025cs.AI cs.CL cs.LGcited by 22

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Existing reasoning benchmarks for large language models (LLMs) frequently fail to capture authentic creativity, often rewarding memorization of previously observed patterns. We address this shortcoming with Sudoku-Bench, a curated benchmark of challenging and unconventional Sudoku variants specifically selected to evaluate creative, multi-step logical reasoning. Sudoku variants form an unusually effective domain for reasoning research: each puzzle introduces unique or subtly interacting constraints, making memorization infeasible and requiring solvers to identify novel logical breakthroughs (``break-ins''). Despite their diversity, Sudoku variants maintain a common and compact structure, enabling clear and consistent evaluation. Sudoku-Bench includes a carefully chosen puzzle set, a standardized text-based puzzle representation, and flexible tools compatible with thousands of publicly available puzzles -- making it easy to extend into a general research environment. Baseline experiments show that state-of-the-art LLMs solve fewer than 15% of puzzles unaided, highlighting significant opportunities to advance long-horizon, strategic reasoning capabilities.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 16 of its 32 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

16 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges13 Feb
  2. 2024Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzles16 Sep
  3. 2024Puzzle Solving using Reasoning of Large Language Models: A Survey17 Feb

Other references · 13

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2025s1: Simple test-time scalingcited by 1,341
  3. 2025Humanity's Last Examcited by 491
  4. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 243
  5. 2023Large Language Model Guided Tree-of-Thoughtcited by 221
  6. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 214
  7. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  8. 2019On the Measure of Intelligencecited by 144
  9. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  10. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  11. 2024Artificial Kuramoto Oscillatory Neuronscited by 38
  12. 2025VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Modelscited by 26
1 more
  1. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Modelscited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 17.

22 papers.

Sort:

By area:

  1. 2025Hierarchical Reasoning Modelcited by 177
  2. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  3. 2025MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluationcited by 12
  4. 2025Reasoning Core: A Scalable RL Environment for LLM Symbolic Reasoningcited by 5
  5. 2026Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoningcited by 5
  6. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  7. 2026Symbol-Equivariant Recurrent Reasoning Modelscited by 4
  8. 2026TopoBench: Benchmarking LLMs on Hard Topological Reasoningcited by 3
  9. 2026Reasoning or Pattern Matching? Probing Large Vision-Language Models with Visual Puzzlescited by 2
  10. 2026Reasoning Core: A Scalable Procedural Data Generation Suite for Symbolic Pre-training and Post-Trainingcited by 2
  11. 2026LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?cited by 1
  12. 2026Training Transformers as a Universal Computercited by 1
10 more
  1. 2026When Recursive Models Finish Computingcited by 1
  2. 2025Toward a Universal Characterization of Sudoku Puzzle Difficultycited by 0
  3. 2026LLMs as ASP Programmers: Self-Correction Enables Task-Agnostic Nonmonotonic Reasoningcited by 0
  4. 2026Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoningcited by 0
  5. 2026G-RRM: Guiding Symbolic Solvers with Recurrent Reasoning Modelscited by 0
  6. 2026Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Modelscited by 0
  7. 2026ALPS: Measuring Valid Creativity in Large Language Models with Mathematical Constructioncited by 0
  8. 2026SMTrap: Cost-Effective DoS Attacks Against Large Reasoning Models via SMT Conflict Guidancecited by 0
  9. 2026Entropy Can Flow, or It Can Guide. Be Entropy. LEDFlow: Introducing Entropy-guided Generation Order into Uniform Discrete Flowcited by 0
  10. 2026AdaC-GRPO: Towards Mastering Long-Horizon Visual Sudoku Puzzles via Adaptive Curriculum Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HardcoreLogic: Challenging Large Reasoning Models with Long-tail Logic Puzzle Games
  2. 2025PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts
  3. 2026MaxSAT-Based Feedback for Guiding Vision-Language Models in Sudoku
  4. 2025Teaching Transformers to Solve Combinatorial Problems through Efficient Trial & Error
  5. 2025MultiZebraLogic: A Multilingual Logical Reasoning Benchmark
  6. 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving
  7. 2026DiBS: Diffusion-Informed Branch Selection
  8. 2025TTT-Bench: A Benchmark for Evaluating Reasoning Ability with Simple and Novel Tic-Tac-Toe-style Games