mlsanityData to 7 Oct 2026

Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

arXiv 2305.04388 ↗First posted 7 May 2023cs.CL cs.AIcited by 1,081

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large Language Models (LLMs) can achieve strong performance on many tasks by producing step-by-step reasoning before giving a final output, often referred to as chain-of-thought reasoning (CoT). It is tempting to interpret these CoT explanations as the LLM's process for solving a task. This level of transparency into LLMs' predictions would yield significant safety benefits. However, we find that CoT explanations can systematically misrepresent the true reason for a model's prediction. We demonstrate that CoT explanations can be heavily influenced by adding biasing features to model inputs--e.g., by reordering the multiple-choice options in a few-shot prompt to make the answer always "(A)"--which models systematically fail to mention in their explanations. When we bias models toward incorrect answers, they frequently generate CoT explanations rationalizing those answers. This causes accuracy to drop by as much as 36% on a suite of 13 tasks from BIG-Bench Hard, when testing with GPT-3.5 from OpenAI and Claude 1.0 from Anthropic. On a social-bias task, model explanations justify giving answers in line with stereotypes without mentioning the influence of these social biases. Our findings indicate that CoT explanations can be plausible yet misleading, which risks increasing our trust in LLMs without guaranteeing their safety. Building more transparent and explainable systems will require either improving CoT faithfulness through targeted efforts or abandoning CoT in favor of alternative methods.

Key facts. Cited by 1,081 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 75 references that are in the map. First posted on arXiv 7 May 2023.

Builds on

43 of 75 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters20 Dec
  2. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning15 Dec
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them17 Oct
  4. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango16 Sep
  5. 2022Large Language Models are Zero-Shot Reasoners24 May
  6. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations24 May
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  6. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  7. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
23 more
  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  2. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  3. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  4. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  5. 2022Self-critiquing models for assisting human evaluatorscited by 372
  6. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  7. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  8. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  9. 2022Language models show human-like content effects on reasoning taskscited by 171
  10. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  11. 2022The Unreliability of Explanations in Few-shot Prompting for Textual Reasoningcited by 169
  12. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  13. 2022Language Models as Agent Modelscited by 131
  14. 2022Faithful Reasoning Using Large Language Modelscited by 130
  15. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  16. 2019Multi-hop Reading Comprehension through Question Decomposition and Rescoringcited by 119
  17. 2020Unsupervised Question Decomposition for Question Answeringcited by 118
  18. 2020Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?cited by 95
  19. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  20. 2022Entailer: Answering Questions with Faithful and Truthful Chains of Reasoningcited by 38
  21. 2022Can Rationalization Improve Robustness?cited by 36
  22. 2022Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Modelcited by 20
  23. 2023Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processescited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 73; 2024: 150; 2025: 260; 2026: 598.

1,081 papers.

Sort:

By area:

  1. 2024OpenAI o1 System Cardcited by 1,220
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  4. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  5. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  6. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  7. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  8. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  9. 2023Explainability for Large Language Models: A Surveycited by 331
  10. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  11. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  12. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
38 more
  1. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 222
  2. 2023An Overview of Catastrophic AI Riskscited by 214
  3. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  4. 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Modelscited by 186
  5. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  6. 2023Gender bias and stereotypes in Large Language Modelscited by 185
  7. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  8. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 166
  9. 2025Phi-4-reasoning Technical Reportcited by 146
  10. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 140
  11. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 132
  12. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  13. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 126
  14. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  15. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  16. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  17. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 118
  18. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  19. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  20. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 108
  21. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
  22. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 107
  23. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 102
  24. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 100
  25. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 97
  26. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  27. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 92
  28. 2023Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Modelscited by 92
  29. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  30. 2024Large Language Models and Games: A Survey and Roadmapcited by 91
  31. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  32. 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
  33. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  34. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  35. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 81
  36. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  37. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 73
  38. 2024A Peek into Token Bias: Large Language Models Are Not Yet Genuine Reasonerscited by 73

50 most cited of 1,081 All 1,081 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Unveiling Confirmation Bias in Chain-of-Thought Reasoning
  2. 2025A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models
  3. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
  4. 2026Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations
  5. 2025Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalization
  6. 2024Chain-of-Thought Reasoning Without Prompting
  7. 2024Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning
  8. 2024The Impact of Reasoning Step Length on Large Language Models