mlsanityData to 7 Oct 2026

Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse

arXiv 2410.21333 ↗First posted 27 Oct 2024cs.LG cs.AI cs.CLcited by 107

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Chain-of-thought (CoT) prompting has become a widely used strategy for improving large language and multimodal model performance. However, it is still an open question under which settings CoT systematically reduces performance. In this paper, we seek to identify the characteristics of tasks where CoT reduces performance by drawing inspiration from cognitive psychology, focusing on six representative tasks from the psychological literature where deliberation hurts performance in humans. In three of these tasks, state-of-the-art models exhibit significant performance drop-offs with CoT (up to 36.3% absolute accuracy for OpenAI o1-preview compared to GPT-4o), while in others, CoT effects are mixed, with positive, neutral, and negative changes. While models and humans do not exhibit perfectly parallel cognitive processes, considering cases where thinking has negative consequences for humans helps identify settings where it negatively impacts models. By connecting the literature on human verbal thinking and deliberation with evaluations of CoT, we offer a perspective for understanding the impact of inference-time reasoning.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 65 references that are in the map. First posted on arXiv 27 Oct 2024.

Builds on

26 of 65 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning18 Sep
  2. 2023Why think step by step? Reasoning emerges from the locality of experience7 Apr
  3. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning15 Dec
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them17 Oct
  5. 2022Automatic Chain of Thought Prompting in Large Language Models7 Oct
  6. 2022Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models16 Sep
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 18

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  5. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  6. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  7. 2022Using cognitive psychology to understand GPT-3cited by 389
  8. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
  9. 2024CogBench: a large language model walks into a psychology labcited by 60
  10. 2024Measuring Implicit Bias in Explicitly Unbiased Large Language Modelscited by 49
  11. 2024Large Language Models Assume People are More Rational than We Really arecited by 49
  12. 2023Large language models predict human sensory judgments across six modalitiescited by 47
6 more
  1. 2020Understanding Human Intelligence through Human Limitationscited by 33
  2. 2024Incoherent Probability Judgments in Large Language Modelscited by 29
  3. 2024How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?cited by 27
  4. 2022Words are all you need? Language as an approximation for human similarity judgmentscited by 18
  5. 2024A Rational Analysis of the Speech-to-Song Illusioncited by 4
  6. 2024Analyzing the Roles of Language and Vision in Learning from Limited Datacited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 61; 2026: 40.

107 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  3. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  4. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 48
  5. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  6. 2025Think or Not Think: A Study of Explicit Thinking in Rule-Based Visual Reinforcement Fine-Tuningcited by 44
  7. 2024Disentangling Memory and Reasoning Ability in Large Language Modelscited by 43
  8. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  9. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  10. 2025OptimalThinkingBench: Evaluating Over and Underthinking in LLMscited by 35
  11. 2025Demystifying Domain-adaptive Post-training for Financial LLMscited by 24
  12. 2025Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completioncited by 22
38 more
  1. 2026Brittlebench: Quantifying LLM robustness via prompt sensitivitycited by 19
  2. 2026LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signalscited by 18
  3. 2025AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Modelscited by 16
  4. 2025Mull-Tokens: Modality-Agnostic Latent Thinkingcited by 15
  5. 2026ProAgentBench: Evaluating LLM Agents for Proactive Assistance with Real-World Datacited by 15
  6. 2025Multimodal LLMs Can Reason about Aesthetics in Zero-Shotcited by 14
  7. 2025Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systemscited by 13
  8. 2025EndoBench: A Comprehensive Evaluation of Multi-Modal Large Language Models for Endoscopy Analysiscited by 13
  9. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 12
  10. 2024Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Fieldcited by 10
  11. 2025Unveiling Confirmation Bias in Chain-of-Thought Reasoningcited by 10
  12. 2025A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experimentscited by 10
  13. 2025Do LLMs Really Need 10+ Thoughts for "Find the Time 1000 Days Later"? Towards Structural Understanding of LLM Overthinkingcited by 10
  14. 2026RubricRAG: Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generationcited by 10
  15. 2025Levels of Analysis for Large Language Modelscited by 9
  16. 2025Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?cited by 9
  17. 2025Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Modelscited by 9
  18. 2026Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Modelscited by 9
  19. 2024Watson: A Cognitive Observability Framework for the Reasoning of LLM-Powered Agentscited by 8
  20. 2025Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?cited by 8
  21. 2026Can Textual Reasoning Improve the Performance of MLLMs on Fine-grained Visual Classification?cited by 8
  22. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomaliescited by 8
  23. 2024Blind Spot Navigation in Large Language Model Reasoning with Thought Space Explorercited by 7
  24. 2025MoVa: Towards Generalizable Classification of Human Morals and Valuescited by 7
  25. 2025Are Large Language Models Sensitive to the Motives Behind Communication?cited by 7
  26. 2025ARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative Refinementcited by 6
  27. 2025DeepSeek-R1 vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and Summarization?cited by 5
  28. 2025AdaptMI: Adaptive Skill-based In-context Math Instruction for Small Language Modelscited by 5
  29. 2025LLM Agents Are Hypersensitive to Nudgescited by 5
  30. 2025Performance of GPT-5 in Brain Tumor MRI Reasoningcited by 5
  31. 2025PersonaFuse: A Personality Activation-Driven Framework for Enhancing Human-LLM Interactionscited by 5
  32. 2025SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversationscited by 5
  33. 2025Chain-of-Thought as a Lens: Evaluating Structured Reasoning Alignment between Human Preferences and Large Language Modelscited by 5
  34. 2025Self-Adaptive Cognitive Debiasing for Large Language Models in Decision-Makingcited by 4
  35. 2025SQUiD: Synthesizing Relational Databases from Unstructured Textcited by 4
  36. 2025Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Modelscited by 4
  37. 2025MotiveBench: How Far Are We From Human-Like Motivational Reasoning in Large Language Models?cited by 4
  38. 2025The Other Mind: How Language Models Exhibit Human Temporal Cognitioncited by 4

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulness
  2. 2026When does Chain-of-Thought Help: A Markovian Perspective
  3. 2025How Chain-of-Thought Works? Tracing Information Flow from Decoding, Projection, and Activation
  4. 2025Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting
  5. 2024Chain of Thought Still Thinks Fast: APriCoT Helps with Thinking Slow
  6. 2024Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning
  7. 2023Towards Better Chain-of-Thought Prompting Strategies: A Survey
  8. 2023Stress Testing Chain-of-Thought Prompting for Large Language Models