mlsanityData to 7 Oct 2026

Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters

arXiv 2212.10001 ↗First posted 20 Dec 2022cs.CLcited by 221

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning abilities of large language models (LLMs). CoT explicitly encourages the LLM to generate intermediate rationales for solving a problem, by providing a series of reasoning steps in the demonstrations. Despite its success, there is still little understanding of what makes CoT prompting effective and which aspects of the demonstrated reasoning steps contribute to its performance. In this paper, we show that CoT reasoning is possible even with invalid demonstrations - prompting with invalid reasoning steps can achieve over 80-90% of the performance obtained using CoT under various metrics, while still generating coherent lines of reasoning during inference. Further experiments show that other aspects of the rationales, such as being relevant to the query and correctly ordering the reasoning steps, are much more important for effective CoT reasoning. Overall, these findings both deepen our understanding of CoT prompting, and open up new questions regarding LLMs' capability to learn to reason in context.

Key facts. Cited by 221 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 40 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

25 of 40 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks22 Nov
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them17 Oct
  3. 2022Automatic Chain of Thought Prompting in Large Language Models7 Oct
  4. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango16 Sep
  5. 2022Large Language Models are Zero-Shot Reasoners24 May
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  12. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
6 more
  1. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  2. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  3. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  4. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  5. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  6. 2022Can Large Language Models Truly Understand Prompts? A Case Study with Negated Promptscited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 57; 2024: 73; 2025: 63; 2026: 27.

221 papers.

Sort:

By area:

  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  3. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  4. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  6. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  7. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  8. 2023Larger language models do in-context learning differentlycited by 386
  9. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  10. 2023Explainability for Large Language Models: A Surveycited by 331
  11. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  12. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
38 more
  1. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  2. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  3. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  4. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  5. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  6. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  7. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  8. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  9. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 126
  10. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  11. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  12. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 106
  13. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  14. 2025How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approachcited by 95
  15. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  16. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 92
  17. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 80
  18. 2023Boosting Language Models Reasoning with Chain-of-Knowledge Promptingcited by 77
  19. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  20. 2024PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generationcited by 77
  21. 2023Eight Things to Know about Large Language Modelscited by 73
  22. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73
  23. 2023Symbol tuning improves in-context learning in language modelscited by 72
  24. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  25. 2023Dr.ICL: Demonstration-Retrieved In-context Learningcited by 71
  26. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 71
  27. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  28. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  29. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  30. 2023Empirical Study of Zero-Shot NER with ChatGPTcited by 60
  31. 2024Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thoughtcited by 56
  32. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  33. 2024The opportunities and risks of large language models in mental healthcited by 54
  34. 2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 53
  35. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  36. 2023How Many Demonstrations Do You Need for In-context Learning?cited by 52
  37. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  38. 2024A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Taskcited by 49

50 most cited of 221 All 221 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns
  2. 2023Stress Testing Chain-of-Thought Prompting for Large Language Models
  3. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
  4. 2024Chain-of-Thought Reasoning Without Prompting
  5. 2024ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting
  6. 2025Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting
  7. 2025Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning
  8. 2024The Impact of Reasoning Step Length on Large Language Models