mlsanityData to 7 Oct 2026

Chain-of-Thought Reasoning In The Wild Is Not Always Faithful

arXiv 2503.08679 ↗First posted 11 Mar 2025cs.AI cs.CL cs.LGcited by 166

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Recent studies indicate that when faced with explicit biases in prompts, models often omit mentioning these biases in their Chain-of-Thought (CoT) output, revealing that verbalized reasoning can give an incorrect picture of how models arrive at conclusions (unfaithfulness). In this work, we show that unfaithful CoT also occurs on naturally worded, non-adversarial prompts without adding artificial biases or editing model outputs. We find that when separately presented with the questions "Is X bigger than Y?" and "Is Y bigger than X?", models sometimes produce superficially coherent arguments to justify systematically answering Yes to both or No to both, despite the contradiction. We present preliminary evidence that this is due to models' implicit biases towards Yes or No, labeling this Implicit Post-Hoc Rationalization. Our results reveal rates up to 13% for production models, and while frontier models are more faithful, none are entirely so, including thinking models like DeepSeek R1 (0.37%) and Sonnet 3.7 with thinking (0.04%). We also investigate Unfaithful Illogical Shortcuts, where models use subtly illogical reasoning to make speculative answers to hard math problems seem rigorously proven. Our findings indicate that while CoT can be useful for assessing outputs, it is not a complete account of the internal process that produced the model's answer and should be used with caution in agentic or safety-critical settings.

Key facts. Cited by 166 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 98 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

25 of 98 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Training Large Language Models to Reason in a Continuous Latent Space9 Dec
  2. 2024Dissociation of Faithful and Unfaithful Reasoning in LLMs23 May
  3. 2024Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought8 Mar
  4. 2023On Measuring Faithfulness or Self-consistency of Natural Language Explanations13 Nov
  5. 2023Preventing Language Models From Hiding Their Reasoning27 Oct
  6. 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
  7. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May
  8. 2023Faithful Chain-of-Thought Reasoning31 Jan
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 16

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  6. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  7. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  8. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  9. 2023Language Models Represent Space and Timecited by 306
  10. 2024PutnamBench: Evaluating Neural Theorem-Provers on the Putnam Mathematical Competitioncited by 153
  11. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 153
  12. 2023Faithfulness Tests for Natural Language Explanationscited by 107
4 more
  1. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 107
  2. 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
  3. 2024The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Modelscited by 29
  4. 2023Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination?cited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 73; 2026: 93.

166 papers.

Sort:

By area:

  1. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  2. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 276
  3. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 222
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  5. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 146
  6. 2025Phi-4-reasoning Technical Reportcited by 146
  7. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  8. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 126
  9. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 63
  10. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  11. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 57
  12. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
38 more
  1. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 49
  2. 2025Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokenscited by 41
  3. 2025Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!cited by 36
  4. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 26
  5. 2025Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effortcited by 25
  6. 2025FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoningcited by 25
  7. 2025Measuring the Faithfulness of Thinking Drafts in Large Reasoning Modelscited by 22
  8. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  9. 2025Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Modelscited by 19
  10. 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 19
  11. 2025Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorabilitycited by 18
  12. 2026LLM Reasoning as Trajectories: Step-Specific Representation Geometry and Correctness Signalscited by 18
  13. 2025Benchmarking Deception Probes via Black-to-White Performance Boostscited by 17
  14. 2025LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoringcited by 17
  15. 2025Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Modelscited by 16
  16. 2025Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbositycited by 15
  17. 2025Thought Branches: Interpreting LLM Reasoning Requires Resamplingcited by 15
  18. 2025Is Chain-of-Thought Really Not Explainability? Chain-of-Thought Can Be Faithful without Hint Verbalizationcited by 15
  19. 2026Counterfactual Simulation Training for Chain-of-Thought Faithfulnesscited by 15
  20. 2025Can Aha Moments Be Fake? Towards Quantifying Decorative and True Thinking in Chain-of-Thoughtcited by 14
  21. 2025When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracycited by 13
  22. 2025Joint Evaluation of Answer and Reasoning Consistency for Hallucination Detection in Large Reasoning Modelscited by 13
  23. 2025The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judgecited by 13
  24. 2026Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMscited by 13
  25. 2025The Geometry of Self-Verification in a Task-Specific Reasoning Modelcited by 12
  26. 2025A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Modelscited by 12
  27. 2026A Positive Case for Faithfulness: LLM Self-Explanations Help Predict Model Behaviorcited by 12
  28. 2025Do Cognitively Interpretable Reasoning Traces Improve LLM Performance?cited by 11
  29. 2025Difficulties with Evaluating a Deception Detector for AIscited by 11
  30. 2025The First Impression Problem: Internal Bias Triggers Overthinking in Reasoning Modelscited by 10
  31. 2025The Dead Salmons of AI Interpretabilitycited by 10
  32. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewardscited by 10
  33. 2025Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanationscited by 9
  34. 2025Reasoning or Retrieval? A Study of Answer Attribution on Large Reasoning Modelscited by 9
  35. 2026RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Modelscited by 9
  36. 2025Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?cited by 8
  37. 2025Do Activation Verbalization Methods Convey Privileged Information?cited by 8
  38. 2025Investigating CoT Monitorability in Large Reasoning Modelscited by 8

50 most cited of 166 All 166 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure
  2. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps
  3. 2026Reasoning-Token Spikes Under Prompted Untruthful Responding in Large Language Models
  4. 2026Does Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters
  5. 2025Unveiling Confirmation Bias in Chain-of-Thought Reasoning
  6. 2026Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations
  7. 2026Do Models Know Why They Changed Their Mind? Interpretability and Faithfulness of Chain-of-Thought Under Knowledge Conflict
  8. 2026Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Reasoning Models?