mlsanityData to 14 Oct 2026

ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning

arXiv 2502.01100 ↗First posted 3 Feb 2025cs.AI cs.CL cs.LGcited by 201

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We investigate the logical reasoning capabilities of large language models (LLMs) and their scalability in complex non-monotonic reasoning. To this end, we introduce ZebraLogic, a comprehensive evaluation framework for assessing LLM reasoning performance on logic grid puzzles derived from constraint satisfaction problems (CSPs). ZebraLogic enables the generation of puzzles with controllable and quantifiable complexity, facilitating a systematic study of the scaling limits of models such as Llama, o1 models, and DeepSeek-R1. By encompassing a broad range of search space complexities and diverse logical constraints, ZebraLogic provides a structured environment to evaluate reasoning under increasing difficulty. Our results reveal a significant decline in accuracy as problem complexity grows -- a phenomenon we term the curse of complexity. This limitation persists even with larger models and increased inference-time computation, suggesting inherent constraints in current LLM reasoning capabilities. Additionally, we explore strategies to enhance logical reasoning, including Best-of-N sampling, backtracking mechanisms, and self-verification prompts. Our findings offer critical insights into the scalability of LLM reasoning, highlight fundamental limitations, and outline potential directions for improvement.

Key facts. Cited by 201 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 19 of its 33 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

19 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024On Memorization of Large Language Models in Logical Reasoning30 Oct
  2. 2024Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoning10 Oct
  3. 2024Can Transformers Reason Logically? A Study in SAT Solving9 Oct
  4. 2024Do Large Language Models Understand Logic or Just Mimick Context?19 Feb
  5. 2022Can Transformers Reason in Fragments of Natural Language?10 Nov

Other references · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  8. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  10. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  11. 2020Transformers as Soft Reasoners over Languagecited by 270
  12. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
2 more
  1. 2021Pushing the Limits of Rule Reasoning in Transformers through Natural Language Satisfiabilitycited by 18
  2. 2024Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter?cited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 91; 2026: 110.

201 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2025Kimi K2: Open Agentic Intelligencecited by 567
  3. 2025Qwen3-Omni Technical Reportcited by 497
  4. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  5. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  6. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  7. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  8. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 166
  9. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  10. 2025Soft Adaptive Policy Optimizationcited by 96
  11. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 93
  12. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 87
38 more
  1. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspectivecited by 76
  2. 2025LongCat-Flash Technical Reportcited by 58
  3. 2025Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzlescited by 56
  4. 2025CodeI/O: Condensing Reasoning Patterns via Code Input-Output Predictioncited by 55
  5. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 55
  6. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 54
  7. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Surveycited by 51
  8. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  9. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  10. 2025SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulascited by 33
  11. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 30
  12. 2025Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thoughtcited by 29
  13. 2025LongCat-Flash-Omni Technical Reportcited by 26
  14. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 25
  15. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  16. 2025Introducing LongCat-Flash-Thinking: A Technical Reportcited by 22
  17. 2026Olmo Hybrid: From Theory to Practice and Backcited by 22
  18. 2025InternBootcamp: Boosting LLM Reasoning with Verifiable Task Scalingcited by 20
  19. 2025Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoningcited by 17
  20. 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learningcited by 17
  21. 2025Generalizable End-to-End Tool-Use RL with Synthetic CodeGymcited by 17
  22. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  23. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 15
  24. 2025GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMscited by 13
  25. 2025Let Me Think! A Long Chain-of-Thought Can Be Worth Exponentially Many Short Onescited by 13
  26. 2025TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Modelscited by 13
  27. 2025MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoEcited by 13
  28. 2025Native Parallel Reasoner: Reasoning in Parallelism via Self-Distilled Reinforcement Learningcited by 12
  29. 2026VERGE: Formal Refinement and Guidance Engine for Verifiable LLM Reasoningcited by 12
  30. 2026DOPD: Dual On-policy Distillationcited by 12
  31. 2025Privacy Reasoning in Ambiguous Contextscited by 11
  32. 2025SAND-Math: Using LLMs to Generate Novel, Difficult and Useful Mathematics Questions and Answerscited by 11
  33. 2025NP-Engine: Empowering Optimization Reasoning in Large Language Models with Verifiable Synthetic NP Problemscited by 11
  34. 2026Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrowcited by 11
  35. 2026ERNIE 5.0 Technical Reportcited by 11
  36. 2026Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVRcited by 11
  37. 2025Do LLMs Really Need 10+ Thoughts for "Find the Time 1000 Days Later"? Towards Structural Understanding of LLM Overthinkingcited by 10
  38. 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 10

50 most cited of 201 All 201 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Effort and Problem Complexity: A Scaling Analysis in LLMs
  2. 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving
  3. 2025MultiZebraLogic: A Multilingual Logical Reasoning Benchmark
  4. 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models
  5. 2024Inference Scaling vs Reasoning: An Empirical Analysis of Compute-Optimal LLM Problem-Solving
  6. 2025A Survey of Scaling in Large Language Model Reasoning
  7. 2026Empirical Evidence of Complexity-Induced Limits in Large Language Models on Finite Discrete State-Space Problems with Explicit Validity Constraints
  8. 2024Puzzle Solving using Reasoning of Large Language Models: A Survey