# The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning

arXiv: https://arxiv.org/abs/2504.05081 · HTML: https://mlsanity.com/paper/2504.05081/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 38 of its 58 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 30 of 38)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [Chain of Thought Empowers Transformers to Solve Inherently Serial Problems](https://mlsanity.com/paper/2402.12875/) (2024)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [When More is Less: Understanding Chain-of-Thought Length in LLMs](https://mlsanity.com/paper/2502.07266/) (2025)
- [The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning](https://mlsanity.com/paper/2205.03401/) (2022)
- [The Impact of Reasoning Step Length on Large Language Models](https://mlsanity.com/paper/2401.04925/) (2024)
- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [LLM-SR: Scientific Equation Discovery via Programming with Large Language Models](https://mlsanity.com/paper/2404.18400/) (2024)
- [Hypothesis Search: Inductive Reasoning with Language Models](https://mlsanity.com/paper/2309.05660/) (2023)
- [Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost](https://mlsanity.com/paper/2407.19825/) (2024)
- [Chain of Thoughtlessness? An Analysis of CoT in Planning](https://mlsanity.com/paper/2405.04776/) (2024)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations](https://mlsanity.com/paper/2305.18354/) (2023)
- [Learning Compositional Rules via Neural Program Synthesis](https://mlsanity.com/paper/2003.05562/) (2020)

## Built on by (most cited first, 30 of 30)

- [ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Data](https://mlsanity.com/paper/2602.04482/) (2026)
- [RoT: Enhancing Table Reasoning with Iterative Row-Wise Traversals](https://mlsanity.com/paper/2505.15110/) (2025)
- [Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models](https://mlsanity.com/paper/2506.07936/) (2025)
- [LLM-Hanabi: Evaluating Multi-Agent Gameplays with Theory-of-Mind and Rationale Inference in Imperfect Information Collaboration Game](https://mlsanity.com/paper/2510.04980/) (2025)
- [Brain-Inspired Graph Multi-Agent Systems for LLM Reasoning](https://mlsanity.com/paper/2603.15371/) (2026)
- [A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search](https://mlsanity.com/paper/2507.00004/) (2025)
- [DixitWorld: Evaluating Multimodal Abductive Reasoning in Vision-Language Models with Multi-Agent Dixit Gameplay](https://mlsanity.com/paper/2510.10117/) (2025)
- [CritiCal: Can Critique Help LLM Uncertainty or Confidence Calibration?](https://mlsanity.com/paper/2510.24505/) (2025)
- [Argument-Based Consistency in Toxicity Explanations of LLMs](https://mlsanity.com/paper/2506.19113/) (2025)
- [Structuring the Unstructured: A Systematic Review of Text-to-Structure Generation for Agentic AI with a Universal Evaluation Framework](https://mlsanity.com/paper/2508.12257/) (2025)
- [When does Chain-of-Thought Help: A Markovian Perspective](https://mlsanity.com/paper/2603.00306/) (2026)
- [Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusions](https://mlsanity.com/paper/2506.23146/) (2025)
- [From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs](https://mlsanity.com/paper/2509.23196/) (2025)
- [Analyzing the Power of Chain of Thought through Memorization Capabilities](https://mlsanity.com/paper/2511.01190/) (2025)
- [Can Small Agents Collaborate to Beat a Single Large Language Model?](https://mlsanity.com/paper/2601.11327/) (2026)
- [Process In-Context Learning: Enhancing Mathematical Reasoning via Dynamic Demonstration Insertion](https://mlsanity.com/paper/2601.11979/) (2026)
- [Scaling Reasoning Hop Exposes Weaknesses: Demystifying and Improving Hop Generalization in Large Language Models](https://mlsanity.com/paper/2601.21214/) (2026)
- [InEdit-Bench: Benchmarking Intermediate Logical Pathways for Intelligent Image Editing Models](https://mlsanity.com/paper/2603.03657/) (2026)
- [Regime-aware financial volatility forecasting via in-context learning](https://mlsanity.com/paper/2603.10299/) (2026)
- [How Clued up are LLMs? Evaluating Multi-Step Deductive Reasoning in a Text-Based Game Environment](https://mlsanity.com/paper/2603.17169/) (2026)
- [When Reasoning Hurts Legal Drafting: The Verbalization Bottleneck in Patent Claim Generation](https://mlsanity.com/paper/2607.10480/) (2026)
- [Understanding Benchmark Language Under Weakened Formal Semantics](https://mlsanity.com/paper/2509.17455/) (2025)
- [Reasoning About Reasoning: Towards Informed and Reflective Use of LLM Reasoning in HCI](https://mlsanity.com/paper/2510.22978/) (2025)
- [A New Strategy for Artificial Intelligence: Training Foundation Models Directly on Human Brain Data](https://mlsanity.com/paper/2601.12053/) (2026)
- [Instruct-ICL: Instruction-Guided In-Context Learning for Post-Disaster Damage Assessment](https://mlsanity.com/paper/2605.11439/) (2026)
- [Can LLM Teams Play What? Where? When?](https://mlsanity.com/paper/2605.30459/) (2026)
- [State commitment learning: training language models to distinguish computation from memory](https://mlsanity.com/paper/2606.05201/) (2026)
- [SciLens: Multi-modal Scientific Claim Verification with Agentic Entailment and Grounding](https://mlsanity.com/paper/2606.20873/) (2026)
- [Different Teachers, Different Capabilities: Sub-1B On-Device Distillation for Structured Text Enrichment](https://mlsanity.com/paper/2607.08268/) (2026)
- [Clueing up LLMs with Tool-Augmented Deductive Reasoning](https://mlsanity.com/paper/2609.18736/) (2026)

## Abstract (arXiv)

Chain-of-Thought (CoT) prompting has been widely recognized for its ability to enhance reasoning capabilities in large language models (LLMs). However, our study reveals a surprising contradiction to this prevailing perspective within the fundamental domain of pattern-based in-context learning (ICL). Through extensive experiments involving 16 state-of-the-art LLMs and nine diverse pattern-based ICL datasets, we demonstrate that CoT and its reasoning variants consistently underperform direct answering across varying model scales and benchmark complexities. To systematically investigate this unexpected phenomenon, we designed extensive experiments to validate several hypothetical explanations. Our analysis uncovers a fundamental hybrid mechanism of explicit-implicit reasoning driving CoT's performance in pattern-based ICL: while explicit reasoning falters due to LLMs' struggles to infer underlying patterns from demonstrations, implicit reasoning-disrupted by the increased contextual distance of CoT rationales-often compensates, delivering correct answers despite flawed rationales. This hybrid mechanism explains CoT's relative underperformance, as noise from weak explicit inference undermines the process, even as implicit mechanisms partially salvage outcomes. Notably, even long-CoT reasoning models, which excel in abstract and symbolic reasoning, fail to fully overcome these limitations despite higher computational costs. Our findings challenge existing assumptions regarding the universal efficacy of CoT, yielding novel insights into its limitations and guiding future research toward more nuanced and effective reasoning methodologies for LLMs.