mlsanityData to 7 Oct 2026

Can Language Models Learn to Skip Steps?

arXiv 2411.01855 ↗First posted 4 Nov 2024cs.CLcited by 59

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Trained on vast corpora of human language, language models demonstrate emergent human-like reasoning abilities. Yet they are still far from true intelligence, which opens up intriguing opportunities to explore the parallels of humans and model behaviors. In this work, we study the ability to skip steps in reasoning - a hallmark of human expertise developed through practice. Unlike humans, who may skip steps to enhance efficiency or to reduce cognitive load, models do not inherently possess such motivations to minimize reasoning steps. To address this, we introduce a controlled framework that stimulates step-skipping behavior by iteratively refining models to generate shorter and accurate reasoning paths. Empirical results indicate that models can develop the step skipping ability under our guidance. Moreover, after fine-tuning on expanded datasets that include both complete and skipped reasoning sequences, the models can not only resolve tasks with increased efficiency without sacrificing accuracy, but also exhibit comparable and even enhanced generalization capabilities in out-of-domain scenarios. Our work presents the first exploration into human-like step-skipping ability and provides fresh perspectives on how such cognitive abilities can benefit AI models.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 39 of its 51 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

39 of 51 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey2 Apr
  2. 2024The Unreasonable Effectiveness of Easy Training Data for Hard Tasks12 Jan
  3. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
  4. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models19 Apr
  5. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  6. 2022Language models show human-like content effects on reasoning tasks14 Jul
  7. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning19 May
  8. 2022Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks11 May

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  9. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  10. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  11. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  12. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
19 more
  1. 2022Using cognitive psychology to understand GPT-3cited by 389
  2. 2022Transformers Learn Shortcuts to Automatacited by 260
  3. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  4. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Stepcited by 225
  5. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  6. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  7. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 165
  8. 2024Distilling System 2 into System 1cited by 152
  9. 2023Teaching Arithmetic to Small Transformerscited by 121
  10. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  11. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  12. 2024CogBench: a large language model walks into a psychology labcited by 60
  13. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  14. 2021Why Machine Reading Comprehension Models Learn Shortcuts?cited by 40
  15. 2023Towards A Holistic Landscape of Situated Theory of Mind in Large Language Modelscited by 32
  16. 2024Weak-to-Strong Reasoningcited by 32
  17. 2023Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-Thoughtscited by 27
  18. 2023Unveiling Theory of Mind in Large Language Models: A Parallel to Single Neurons in the Human Braincited by 18
  19. 2021Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 45; 2026: 13.

59 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 246
  3. 2025Dynamic Early Exit in Reasoning Modelscited by 217
  4. 2025Reasoning Models Can Be Effective Without Thinkingcited by 182
  5. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 167
  6. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  7. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  8. 2025Efficient Reasoning Models: A Surveycited by 93
  9. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
  10. 2025AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learningcited by 86
  11. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  12. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 53
38 more
  1. 2025AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficulty-Aware Budgetingcited by 50
  2. 2025SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoningcited by 47
  3. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  4. 2025Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropycited by 37
  5. 2025THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating Overthinking in Reasoning Modelscited by 34
  6. 2025DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learningcited by 34
  7. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  8. 2025Entropy-based Exploration Conduction for Multi-step Reasoningcited by 28
  9. 2025Shorten After You're Right: Lazy Length Penalties for Reasoning RLcited by 25
  10. 2025VeriThinker: Learning to Verify Makes Reasoning Model Efficientcited by 24
  11. 2025Thought Manipulation: External Thought Can Be Efficient for Large Reasoning Modelscited by 23
  12. 2025System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcutscited by 23
  13. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  14. 2025KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoningcited by 19
  15. 2025OThink-R1: Intrinsic Fast/Slow Thinking Mode Switching for Over-Reasoning Mitigationcited by 17
  16. 2026Does Your Reasoning Model Implicitly Know When to Stop Thinking?cited by 17
  17. 2025Fast on the Easy, Deep on the Hard: Efficient Reasoning via Powered Length Penaltycited by 15
  18. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  19. 2025Learning to Focus: Causal Attention Distillation via Gradient-Guided Token Pruningcited by 13
  20. 2025AutoL2S: Auto Long-Short Reasoning for Efficient Large Language Modelscited by 12
  21. 2025Self-Route: Automatic Mode Switching via Capability Estimation for Efficient Reasoningcited by 8
  22. 2025Long or short CoT? Investigating Instance-level Switch of Large Reasoning Modelscited by 8
  23. 2025How Far Are We from Optimal Reasoning Efficiency?cited by 8
  24. 2025Exploring and Exploiting the Inherent Efficiency within Large Reasoning Models for Self-Guided Efficiency Enhancementcited by 8
  25. 2026FoE: Forest of Errors Makes the First Solution the Best in Large Reasoning Modelscited by 8
  26. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 7
  27. 2025Are Large Reasoning Models Interruptible?cited by 6
  28. 2025Fast Quiet-STaR: Thinking Without Thought Tokenscited by 5
  29. 2025PREMISE: Scalable and Strategic Prompt Optimization for Efficient Mathematical Reasoning in Large Modelscited by 5
  30. 2025TokenSqueeze: Performance-Preserving Compression for Reasoning LLMscited by 5
  31. 2024Verbosity-Aware Rationale Reduction: Effective Reduction of Redundant Rationale via Principled Criteriacited by 4
  32. 2025HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMscited by 4
  33. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  34. 2025DEPO: Dual-Efficiency Preference Optimization for LLM Agentscited by 4
  35. 2025Assembly of Experts: Linear-time construction of the Chimera LLM variants with emergent and adaptable behaviorscited by 3
  36. 2025From Long to Lean: Performance-aware and Adaptive Chain-of-Thought Compression via Multi-round Refinementcited by 2
  37. 2026Unlocking the Black Box of Latent Reasoning: An Interpretability-Guided Approach to Interventioncited by 2
  38. 2025Uncovering the Computational Ingredients of Human-Like Representations in LLMscited by 1

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Competence, Shrinking Reasoning: Cognitive Signatures in Language Model Learning
  2. 2025Reasoning Strategies in Large Language Models: Can They Follow, Prefer, and Optimize?
  3. 2025Reasoning Capabilities of Large Language Models on Dynamic Tasks
  4. 2023Studying and improving reasoning in humans and machines
  5. 2026Reasoning aligns language models to human cognition
  6. 2025Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models
  7. 2025Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
  8. 2023Why think step by step? Reasoning emerges from the locality of experience