mlsanityData to 7 Oct 2026

Puzzle Solving using Reasoning of Large Language Models: A Survey

arXiv 2402.11291 ↗First posted 17 Feb 2024cs.CL cs.AIcited by 44

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Exploring the capabilities of Large Language Models (LLMs) in puzzle solving unveils critical insights into their potential and challenges in AI, marking a significant step towards understanding their applicability in complex reasoning tasks. This survey leverages a unique taxonomy -- dividing puzzles into rule-based and rule-less categories -- to critically assess LLMs through various methodologies, including prompting techniques, neuro-symbolic approaches, and fine-tuning. Through a critical review of relevant datasets and benchmarks, we assess LLMs' performance, identifying significant challenges in complex puzzle scenarios. Our findings highlight the disparity between LLM capabilities and human-like reasoning, particularly in those requiring advanced logical inference. The survey underscores the necessity for novel strategies and richer datasets to advance LLMs' puzzle-solving proficiency and contribute to AI's logical reasoning and creative problem-solving advancements.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 64 of its 102 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

64 of 102 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Language Models can be Logical Solvers10 Nov
  2. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers23 Oct
  3. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models2 Oct
  4. 2023LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles21 Aug
  5. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information13 Jun
  6. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples24 May
  7. 2023CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering24 May
  8. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
  9. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-47 Apr
  10. 2022True Detective: A Deep Abductive Reasoning Benchmark Undoable for GPT-3 and Challenging for GPT-420 Dec
  11. 2022Down and Across: Introducing Crossword-Solving as a New NLP Benchmark20 May
  12. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning19 May

Other references · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
38 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  5. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  8. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  9. 2022A Survey on In-context Learningcited by 717
  10. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  11. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  12. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  13. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  14. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  15. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  16. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  17. 2023Large Language Model Guided Tree-of-Thoughtcited by 218
  18. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  19. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 102
  20. 2023LLMs cannot find reasoning errors, but can correct them given the error locationcited by 85
  21. 2022Language Models Can Teach Themselves to Program Bettercited by 75
  22. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 73
  23. 2023Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4cited by 67
  24. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  25. 2023LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplaycited by 62
  26. 2021RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledgecited by 60
  27. 2023Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thoughtcited by 41
  28. 2023Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Codecited by 31
  29. 2023Are ChatGPT and GPT-4 Good Poker Players? -- A Pre-Flop Analysiscited by 20
  30. 2023Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs' Non-linear Thinkingcited by 19
  31. 2021Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLPcited by 16
  32. 2018Solving the Rubik's Cube Without Human Knowledgecited by 15
  33. 2021Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Languagecited by 15
  34. 2023Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Studycited by 14
  35. 2021BiRdQA: A Bilingual Dataset for Question Answering on Tricky Riddlescited by 10
  36. 2021Puzzle Solving without Search or Human Knowledge: An Unnatural Language Approachcited by 8
  37. 2022CC-Riddle: A Question Answering Dataset of Chinese Character Riddlescited by 6
  38. 2023ArabIcros: AI-Powered Arabic Crossword Puzzle Generation for Educational Applicationscited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 23; 2026: 6.

44 papers.

Sort:

By area:

  1. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  2. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  3. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillationcited by 71
  4. 2024Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzlescited by 35
  5. 2025SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulascited by 30
  6. 2024Easy Problems That LLMs Get Wrongcited by 29
  7. 2025Sudoku-Bench: Evaluating creative reasoning with Sudoku variantscited by 21
  8. 2024The Role of Deductive and Inductive Reasoning in Large Language Modelscited by 20
  9. 2025InternBootcamp: Boosting LLM Reasoning with Verifiable Task Scalingcited by 19
  10. 2024Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter?cited by 18
  11. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
  12. 2025Synthelite: Chemist-aligned and feasibility-aware synthesis planning with LLMscited by 10
32 more
  1. 2024Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Modelscited by 9
  2. 2024Zero-shot Model-based Reinforcement Learning using Large Language Modelscited by 6
  3. 2024Attack-in-the-Chain: Bootstrapping Large Language Models for Attacks Against Black-box Neural Ranking Modelscited by 6
  4. 2025Beyond In-Distribution Success: Scaling Curves of CoT Granularity for Language Model Generalizationcited by 5
  5. 2025Explaining Puzzle Solutions in Natural Language: An Exploratory Study on 6x6 Sudokucited by 5
  6. 2025LogicPuzzleRL: Cultivating Robust Mathematical Reasoning in LLMs via Reinforcement Learningcited by 5
  7. 2025The Jumping Reasoning Curve? Tracking the Evolution of Reasoning Performance in GPT-[n] and o-[n] Models on Multimodal Puzzlescited by 4
  8. 2025LR^2Bench: Evaluating Long-chain Reflective Reasoning Capabilities of Large Language Models via Constraint Satisfaction Problemscited by 4
  9. 2026Pencil Puzzle Bench: A Benchmark for Multi-Step Verifiable Reasoningcited by 4
  10. 2024RISCORE: Enhancing In-Context Riddle Solving in Language Models through Context-Reconstructed Example Augmentationcited by 3
  11. 2026TopoBench: Benchmarking LLMs on Hard Topological Reasoningcited by 3
  12. 2024Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebusescited by 2
  13. 2024Enhancing adversarial robustness in Natural Language Inference using explanationscited by 2
  14. 2025Large Language Models in Numberland: A Quick Test of Their Numerical Reasoning Abilitiescited by 2
  15. 2025Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Modelscited by 2
  16. 2025Can LLMs Generate and Solve Linguistic Olympiad Puzzles?cited by 2
  17. 2024AILS-NTUA at SemEval-2024 Task 9: Cracking Brain Teasers: Transformer Models for Lateral Thinking Puzzlescited by 1
  18. 2025Pitfalls of Scale: Investigating the Inverse Task of Redefinition in Large Language Modelscited by 1
  19. 2025LLM-based Automated Theorem Proving Hinges on Scalable Synthetic Data Generationcited by 1
  20. 2025Evaluating Contrast Localizer for Identifying Causal Units in Social & Mathematical Tasks in Language Modelscited by 1
  21. 2025The Riddle of Reflection: Evaluating Reasoning and Self-Awareness in Multilingual LLMs using Indian Riddlescited by 1
  22. 2026Evaluating Robustness of Reasoning Models on Parameterized Logical Problemscited by 1
  23. 2024Harmonic Reasoning in Large Language Modelscited by 0
  24. 2024QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMscited by 0
  25. 2025FSLI: An Interpretable Formal Semantic System for One-Dimensional Ordering Inferencecited by 0
  26. 2025Graph Neural Networks Based Analog Circuit Link Predictioncited by 0
  27. 2025Explaining Hitori Puzzles: Neurosymbolic Proof Staging for Sequential Decisionscited by 0
  28. 2025PHANTOM RECALL: When Familiar Puzzles Fool Smart Modelscited by 0
  29. 2025Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Modelscited by 0
  30. 2025When Do Symbolic Solvers Enhance Reasoning in Large Language Models?cited by 0
  31. 2026AILS-NTUA at SemEval-2026 Task 8: Evaluating Multi-Turn RAG Conversationscited by 0
  32. 2026SMART: A Machine Learning and Monte Carlo Framework for Rapid Analysis of Stochastic Transistor Aging and Process Variation in Digital Circuitscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Logic-of-Thought: Empowering Large Language Models with Logic Programs for Solving Puzzles in Natural Language
  2. 2025Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles
  3. 2025EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges
  4. 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving
  5. 2022Towards Reasoning in Large Language Models: A Survey
  6. 2025Baba is LLM: Reasoning in a Game with Dynamic Rules
  7. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles
  8. 2025Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey