mlsanityData to 7 Oct 2026

To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning

arXiv 2409.12183 ↗First posted 18 Sep 2024cs.CL cs.AI cs.LGcited by 257

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs). But for what kinds of tasks is this extra ``thinking'' really helpful? To analyze this, we conducted a quantitative meta-analysis covering over 100 papers using CoT and ran our own evaluations of 20 datasets across 14 models. Our results show that CoT gives strong performance benefits primarily on tasks involving math or logic, with much smaller gains on other types of tasks. On MMLU, directly generating the answer without CoT leads to almost identical accuracy as CoT unless the question or model's response contains an equals sign, indicating symbolic operations and reasoning. Following this finding, we analyze the behavior of CoT on these problems by separating planning and execution and comparing against tool-augmented LLMs. Much of CoT's gain comes from improving symbolic execution, but it underperforms relative to using a symbolic solver. Our results indicate that CoT can be applied selectively, maintaining performance while saving inference costs. Furthermore, they suggest a need to move beyond prompt-based CoT to new paradigms that better leverage intermediate computation across the whole range of LLM applications.

Key facts. Cited by 257 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 85 of its 103 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

85 of 103 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Faithful Logical Reasoning via Symbolic Chain-of-Thought28 May
  2. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step23 May
  3. 2024Chain of Thoughtlessness? An Analysis of CoT in Planning8 May
  4. 2024On the Empirical Complexity of Reasoning and Planning in LLMs17 Apr
  5. 2023Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models9 Oct
  6. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Models18 Aug
  7. 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
  8. 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents23 May
  9. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models6 May
  10. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters20 Dec
  11. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks22 Nov
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them17 Oct
6 more
  1. 2022Complexity-Based Prompting for Multi-Step Reasoning3 Oct
  2. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango16 Sep
  3. 2022Large Language Models are Zero-Shot Reasoners24 May
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 67

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2023Let's Verify Step by Stepcited by 3,598
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  12. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  13. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  14. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  15. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  16. 2023Large Language Models as Optimizerscited by 830
  17. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  18. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  19. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  20. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  21. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  22. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  23. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
  24. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  25. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  26. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308
  27. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  28. 2022Transformers Learn Shortcuts to Automatacited by 260
  29. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  30. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 195
  31. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
  32. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  33. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  34. 2024On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Taskscited by 137
  35. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  36. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  37. 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
  38. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 157; 2026: 73.

257 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  3. 2024Alignment faking in large language modelscited by 316
  4. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuningcited by 167
  5. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  6. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  7. 2025Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketchingcited by 116
  8. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  9. 2025SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMscited by 110
  10. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  11. 2024Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worsecited by 107
  12. 2025Self-Training Elicits Concise Reasoning in Large Language Modelscited by 97
38 more
  1. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  2. 2025Efficient Reasoning Models: A Surveycited by 93
  3. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  4. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  5. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  6. 2025Reasoning Models Better Express Their Confidencecited by 63
  7. 2024Interpretable Contrastive Monte Carlo Tree Search Reasoningcited by 62
  8. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  9. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  10. 2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 54
  11. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewardingcited by 53
  12. 2025Training Strategies for Efficient Embodied Reasoningcited by 53
  13. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 52
  14. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  15. 2025When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMscited by 51
  16. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 48
  17. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  18. 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoningcited by 45
  19. 2025Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuningcited by 44
  20. 2024Disentangling Memory and Reasoning Ability in Large Language Modelscited by 43
  21. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 43
  22. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 41
  23. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  24. 2025VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Datacited by 39
  25. 2025SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoningcited by 38
  26. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  27. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  28. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  29. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 32
  30. 2025Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lenscited by 32
  31. 2024A Picture is Worth A Thousand Numbers: Enabling LLMs Reason about Time Series via Visualizationcited by 31
  32. 2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
  33. 2025The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learningcited by 30
  34. 2025A Survey of Optimization Modeling Meets LLMs: Progress and Future Directionscited by 30
  35. 2024SPRIG: Improving Large Language Model Performance by System Prompt Optimizationcited by 28
  36. 2025Entropy-based Exploration Conduction for Multi-step Reasoningcited by 28
  37. 2025Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perceptioncited by 26
  38. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25

50 most cited of 257 All 257 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve
  2. 2023Stress Testing Chain-of-Thought Prompting for Large Language Models
  3. 2024Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning
  4. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
  5. 2025Non-Interactive Symbolic-Aided Chain-of-Thought for Logical Reasoning
  6. 2023Faithful Chain-of-Thought Reasoning
  7. 2025Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
  8. 2026The Potential of CoT for Reasoning: A Closer Look at Trace Dynamics