# Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting

arXiv: https://arxiv.org/abs/2506.07142 · HTML: https://mlsanity.com/paper/2506.07142/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 2 of its 3 references that are in the map.
- First posted on arXiv 8 Jun 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 2 of 2)

- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations](https://mlsanity.com/paper/2411.00640/) (2024)

## Built on by (most cited first, 13 of 13)

- [Performance of GPT-5 in Brain Tumor MRI Reasoning](https://mlsanity.com/paper/2508.10865/) (2025)
- [Tacit Coordination of Large Language Models](https://mlsanity.com/paper/2601.22184/) (2026)
- [CircuitLM: A Multi-Agent LLM-Aided Design Framework for Generating Circuit Schematics from Natural Language Prompts](https://mlsanity.com/paper/2601.04505/) (2026)
- [The Six Sigma Agent: Achieving Enterprise-Grade Reliability in LLM Systems Through Consensus-Driven Decomposed Execution](https://mlsanity.com/paper/2601.22290/) (2026)
- [Beyond "Hallucinations": A Framework for Stable Human-AI Reasoning](https://mlsanity.com/paper/2510.14665/) (2025)
- [Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning](https://mlsanity.com/paper/2602.21103/) (2026)
- [Multi-Agent Dialectical Refinement for Enhanced Argument Classification](https://mlsanity.com/paper/2603.27451/) (2026)
- [Paraphrase Brittleness in Production Retrieval-Augmented Commercial Recommendation: Reproducibility Below the Rerun-Stability Baseline](https://mlsanity.com/paper/2605.27440/) (2026)
- [Prompting Science Report 3: I'll pay you or I'll kill you -- but will you care?](https://mlsanity.com/paper/2508.00614/) (2025)
- [Prompting Science Report 4: Playing Pretend: Expert Personas Don't Improve Factual Accuracy](https://mlsanity.com/paper/2512.05858/) (2025)
- [Populism Meets AI: Advancing Populism Research with LLMs](https://mlsanity.com/paper/2510.07458/) (2025)
- [Human-in-the-Loop Control of Objective Drift in LLM-Assisted Computer Science Education](https://mlsanity.com/paper/2604.00281/) (2026)
- [Temperature-Dependent Performance of Prompting Strategies in Extended Reasoning Large Language Models](https://mlsanity.com/paper/2604.08563/) (2026)

## Abstract (arXiv)

This is the second in a series of short reports that seek to help business, education, and policy leaders understand the technical details of working with AI through rigorous testing. In this report, we investigate Chain-of-Thought (CoT) prompting, a technique that encourages a large language model (LLM) to "think step by step" (Wei et al., 2022). CoT is a widely adopted method for improving reasoning tasks, however, our findings reveal a more nuanced picture of its effectiveness. We demonstrate two things: - The effectiveness of Chain-of-Thought prompting can vary greatly depending on the type of task and model. For non-reasoning models, CoT generally improves average performance by a small amount, particularly if the model does not inherently engage in step-by-step processing by default. However, CoT can introduce more variability in answers, sometimes triggering occasional errors in questions the model would otherwise get right. We also found that many recent models perform some form of CoT reasoning even if not asked; for these models, a request to perform CoT had little impact. Performing CoT generally requires far more tokens (increasing cost and time) than direct answers. - For models designed with explicit reasoning capabilities, CoT prompting often results in only marginal, if any, gains in answer accuracy. However, it significantly increases the time and tokens needed to generate a response.