mlsanityData to 14 Oct 2026

The Impact of Reasoning Step Length on Large Language Models

arXiv 2401.04925 ↗First posted 10 Jan 2024cs.CL cs.AIcited by 153

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Chain of Thought (CoT) is significant in improving the reasoning abilities of large language models (LLMs). However, the correlation between the effectiveness of CoT and the length of reasoning steps in prompts remains largely unknown. To shed light on this, we have conducted several empirical experiments to explore the relations. Specifically, we design experiments that expand and compress the rationale reasoning steps within CoT demonstrations while keeping all other factors constant. We have the following key findings. First, the results indicate that lengthening the reasoning steps in prompts, even without adding new information into the prompt, considerably enhances LLMs' reasoning abilities across multiple datasets. Alternatively, shortening the reasoning steps, even while preserving the key information, significantly diminishes the reasoning abilities of models. This finding highlights the importance of the number of steps in CoT prompts and provides practical guidance to make better use of LLMs' potential in complex problem-solving scenarios. Second, we also investigated the relationship between the performance of CoT and the rationales used in demonstrations. Surprisingly, the result shows that even incorrect rationales can yield favorable outcomes if they maintain the requisite length of inference. Third, we observed that the advantages of increasing reasoning steps are task-dependent: simpler tasks require fewer steps, whereas complex tasks gain significantly from longer inference sequences. The code is available at https://github.com/MingyuJ666/The-Impact-of-Reasoning-Step-Length-on-Large-Language-Models

Key facts. Cited by 153 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 26 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

19 of 26 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-based Feature Attributions25 Jul
  2. 2023Dissecting Chain-of-Thought: Compositionality through In-Context Filtering and Learning30 May
  3. 2023Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models1 Feb
  4. 2022Automatic Chain of Thought Prompting in Large Language Models7 Oct
  5. 2022Complexity-Based Prompting for Multi-Step Reasoning3 Oct
  6. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango16 Sep
  7. 2022Large Language Models are Zero-Shot Reasoners24 May
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 10

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  6. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  7. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  8. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  9. 2016Solving General Arithmetic Word Problemscited by 360
  10. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasonerscited by 63

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 48; 2025: 85; 2026: 20.

153 papers.

Sort:

By area:

  1. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  2. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  3. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 369
  4. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  5. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMscited by 249
  6. 2025Training Language Models to Reason Efficientlycited by 247
  7. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 183
  8. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 152
  9. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 145
  10. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  11. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMscited by 117
  12. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 113
38 more
  1. 2024Improving Multi-Agent Debate with Sparse Communication Topologycited by 107
  2. 2025Self-Training Elicits Concise Reasoning in Large Language Modelscited by 99
  3. 2025How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approachcited by 97
  4. 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 96
  5. 2025Efficient Reasoning Models: A Surveycited by 95
  6. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 89
  7. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 88
  8. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 84
  9. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  10. 2025Concise Reasoning via Reinforcement Learningcited by 79
  11. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  12. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 73
  13. 2024Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Samplingcited by 63
  14. 2024Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thoughtcited by 56
  15. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  16. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  17. 2024MoralBench: Moral Evaluation of LLMscited by 50
  18. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  19. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shapingcited by 50
  20. 2024Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilitiescited by 48
  21. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 47
  22. 2024Disentangling Memory and Reasoning Ability in Large Language Modelscited by 44
  23. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agentscited by 42
  24. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longercited by 41
  25. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoningcited by 40
  26. 2024Knowledge Graph Large Language Model (KG-LLM) for Link Predictioncited by 36
  27. 2025OmniRouter: Budget and Performance Controllable Multi-LLM Routingcited by 35
  28. 2024How Likely Do LLMs with CoT Mimic Human Reasoning?cited by 34
  29. 2025Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lenscited by 34
  30. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 33
  31. 2025Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agentscited by 33
  32. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  33. 2024GraphWiz: An Instruction-Following Language Model for Graph Problemscited by 32
  34. 2024Break the Chain: Large Language Models Can be Shortcut Reasonerscited by 32
  35. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  36. 2025The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learningcited by 30
  37. 2025Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimizationcited by 29
  38. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29

50 most cited of 153 All 153 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Stress Testing Chain-of-Thought Prompting for Large Language Models
  2. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
  3. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning
  4. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
  5. 2024Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning
  6. 2025Long Chain-of-Thought Reasoning Across Languages
  7. 2024Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns
  8. 2025Short-Path Prompting in LLMs: Analyzing Reasoning Instability and Solutions for Robust Performance