# Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse

arXiv: https://arxiv.org/abs/2410.21333 · HTML: https://mlsanity.com/paper/2410.21333/

## Key facts

- Cited by 109 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 26 of its 65 references that are in the map.
- First posted on arXiv 27 Oct 2024.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 26 of 26)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2410.05229/) (2024)
- [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022)
- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://mlsanity.com/paper/2212.08061/) (2022)
- [LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks](https://mlsanity.com/paper/2402.01817/) (2024)
- [Why think step by step? Reasoning emerges from the locality of experience](https://mlsanity.com/paper/2304.03843/) (2023)
- [CogBench: a large language model walks into a psychology lab](https://mlsanity.com/paper/2402.18225/) (2024)
- [Large Language Models Assume People are More Rational than We Really are](https://mlsanity.com/paper/2406.17055/) (2024)
- [Measuring Implicit Bias in Explicitly Unbiased Large Language Models](https://mlsanity.com/paper/2402.04105/) (2024)
- [Large language models predict human sensory judgments across six modalities](https://mlsanity.com/paper/2302.01308/) (2023)
- [Understanding Human Intelligence through Human Limitations](https://mlsanity.com/paper/2009.14050/) (2020)
- [Incoherent Probability Judgments in Large Language Models](https://mlsanity.com/paper/2401.16646/) (2024)
- [How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?](https://mlsanity.com/paper/2402.07282/) (2024)
- [Words are all you need? Language as an approximation for human similarity judgments](https://mlsanity.com/paper/2206.04105/) (2022)
- [Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models](https://mlsanity.com/paper/2209.08141/) (2022)
- [A Rational Analysis of the Speech-to-Song Illusion](https://mlsanity.com/paper/2402.06992/) (2024)
- [Analyzing the Roles of Language and Vision in Learning from Limited Data](https://mlsanity.com/paper/2403.19669/) (2024)

## Built on by (most cited first, 30 of 109)

- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems](https://mlsanity.com/paper/2504.09037/) (2025)
- [LLM Social Simulations Are a Promising Research Method](https://mlsanity.com/paper/2504.02234/) (2025)
- [SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors](https://mlsanity.com/paper/2510.17516/) (2025)
- [Crosslingual Reasoning through Test-Time Scaling](https://mlsanity.com/paper/2505.05408/) (2025)
- [Disentangling Memory and Reasoning Ability in Large Language Models](https://mlsanity.com/paper/2411.13504/) (2024)
- [Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuning](https://mlsanity.com/paper/2503.16188/) (2025)
- [Improving LLM-as-a-Judge Inference with the Judgment Distribution](https://mlsanity.com/paper/2503.03064/) (2025)
- [The Homogenizing Effect of Large Language Models on Human Expression and Thought](https://mlsanity.com/paper/2508.01491/) (2025)
- [OptimalThinkingBench: Evaluating Over and Underthinking in LLMs](https://mlsanity.com/paper/2508.13141/) (2025)
- [Demystifying Domain-adaptive Post-training for Financial LLMs](https://mlsanity.com/paper/2501.04961/) (2025)
- [LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signals](https://mlsanity.com/paper/2604.05655/) (2026)
- [Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion](https://mlsanity.com/paper/2505.24362/) (2025)
- [Brittlebench: Quantifying LLM robustness via prompt sensitivity](https://mlsanity.com/paper/2603.13285/) (2026)
- [Mull-Tokens: Modality-Agnostic Latent Thinking](https://mlsanity.com/paper/2512.10941/) (2025)
- [AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Models](https://mlsanity.com/paper/2505.17312/) (2025)
- [ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Data](https://mlsanity.com/paper/2602.04482/) (2026)
- [Multimodal LLMs Can Reason about Aesthetics in Zero-Shot](https://mlsanity.com/paper/2501.09012/) (2025)
- [Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems](https://mlsanity.com/paper/2505.17968/) (2025)
- [EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysis](https://mlsanity.com/paper/2505.23601/) (2025)
- [Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs](https://mlsanity.com/paper/2502.16435/) (2025)
- [A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments](https://mlsanity.com/paper/2509.25609/) (2025)
- [RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation](https://mlsanity.com/paper/2603.20882/) (2026)
- [Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field](https://mlsanity.com/paper/2412.08258/) (2024)
- [Levels of Analysis for Large Language Models](https://mlsanity.com/paper/2503.13401/) (2025)
- [Unveiling Confirmation Bias in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2506.12301/) (2025)
- [Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?](https://mlsanity.com/paper/2506.19467/) (2025)
- [Do LLMs Really Need 10+ Thoughts for "Find the Time 1000 Days Later"? Towards Structural Understanding of LLM Overthinking](https://mlsanity.com/paper/2510.07880/) (2025)
- [Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models](https://mlsanity.com/paper/2511.07979/) (2025)
- [Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?](https://mlsanity.com/paper/2508.19827/) (2025)

## Abstract (arXiv)

Chain-of-thought (CoT) prompting has become a widely used strategy for improving large language and multimodal model performance. However, it is still an open question under which settings CoT systematically reduces performance. In this paper, we seek to identify the characteristics of tasks where CoT reduces performance by drawing inspiration from cognitive psychology, focusing on six representative tasks from the psychological literature where deliberation hurts performance in humans. In three of these tasks, state-of-the-art models exhibit significant performance drop-offs with CoT (up to 36.3% absolute accuracy for OpenAI o1-preview compared to GPT-4o), while in others, CoT effects are mixed, with positive, neutral, and negative changes. While models and humans do not exhibit perfectly parallel cognitive processes, considering cases where thinking has negative consequences for humans helps identify settings where it negatively impacts models. By connecting the literature on human verbal thinking and deliberation with evaluations of CoT, we offer a perspective for understanding the impact of inference-time reasoning.