mlsanityData to 14 Oct 2026

Reasoning Models Don't Always Say What They Think

arXiv 2505.05410 ↗First posted 8 May 2025cs.CL cs.AI cs.LGcited by 415

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Chain-of-thought (CoT) offers a potential boon for AI safety as it allows monitoring a model's CoT to try to understand its intentions and reasoning processes. However, the effectiveness of such monitoring hinges on CoTs faithfully representing models' actual reasoning processes. We evaluate CoT faithfulness of state-of-the-art reasoning models across 6 reasoning hints presented in the prompts and find: (1) for most settings and models tested, CoTs reveal their usage of hints in at least 1% of examples where they use the hint, but the reveal rate is often below 20%, (2) outcome-based reinforcement learning initially improves faithfulness but plateaus without saturating, and (3) when reinforcement learning increases how frequently hints are used (reward hacking), the propensity to verbalize them does not increase, even without training against a CoT monitor. These results suggest that CoT monitoring is a promising way of noticing undesired behaviors during training and evaluations, but that it is not sufficient to rule them out. They also suggest that in settings like ours where CoT reasoning is not necessary, test-time monitoring of CoTs is unlikely to reliably catch rare and catastrophic unexpected behaviors.

Key facts. Cited by 415 papers in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 43 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

37 of 43 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation14 Mar
  2. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithful11 Mar
  3. 2024Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought8 Mar
  4. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning21 Feb
  5. 2023Contrastive Chain-of-Thought Prompting15 Nov
  6. 2023Preventing Language Models From Hiding Their Reasoning27 Oct
  7. 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul
  8. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May
  9. 2023Faithful Chain-of-Thought Reasoning31 Jan
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar

Other references · 27

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
  6. 2022Large Language Models Can Self-Improvecited by 664
  7. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  8. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  9. 2024Alignment faking in large language modelscited by 360
  10. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  11. 2024Frontier Models are Capable of In-context Schemingcited by 287
  12. 2024Iterative Reasoning Preference Optimizationcited by 246
15 more
  1. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 190
  2. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  3. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 108
  4. 2023Question Decomposition Improves the Faithfulness of Model-Generated Reasoningcited by 108
  5. 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
  6. 2024Prover-Verifier Games improve legibility of LLM outputscited by 85
  7. 2025Auditing language models for hidden objectivescited by 83
  8. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 72
  9. 2019An Evaluation of the Human-Interpretability of Explanationcited by 71
  10. 2018Faithful Multimodal Explanation for Visual Question Answeringcited by 64
  11. 2017The Promise and Peril of Human Evaluation for Model Interpretabilitycited by 52
  12. 2018Do Explanations make VQA Models more Predictable to a Human?cited by 52
  13. 2024Safety Cases: How to Justify the Safety of Advanced AI Systemscited by 48
  14. 2024Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuningcited by 26
  15. 2024Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatabilitycited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 138; 2026: 277.

415 papers.

Sort:

By area:

  1. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 302
  2. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 249
  3. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 158
  4. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  5. 2025GRIT: Teaching MLLMs to Think with Imagescited by 125
  6. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
  7. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoningcited by 78
  8. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agentscited by 71
  9. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 61
  10. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  11. 2025When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Modelscited by 50
  12. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lenscited by 43
38 more
  1. 2025Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokenscited by 41
  2. 2025Answer Convergence as a Signal for Early Stopping in Reasoningcited by 40
  3. 2026Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thoughtcited by 40
  4. 2025Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzlescited by 36
  5. 2025Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspectivecited by 34
  6. 2025Enabling Agents to Communicate Entirely in Latent Spacecited by 34
  7. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  8. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 31
  9. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Usecited by 31
  10. 2025Training Language Models to Explain Their Own Computationscited by 30
  11. 2026This human study did not involve human subjects: Validating LLM simulations as behavioral evidencecited by 29
  12. 2025Accumulating Context Changes the Beliefs of Language Modelscited by 28
  13. 2025Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effortcited by 27
  14. 2025FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoningcited by 26
  15. 2025Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisionscited by 25
  16. 2025What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoTcited by 25
  17. 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomescited by 23
  18. 2025Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorabilitycited by 22
  19. 2025Training LLMs for Honesty via Confessionscited by 22
  20. 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 21
  21. 2025WebGuard: Building a Generalizable Guardrail for Web Agentscited by 21
  22. 2026Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representationcited by 21
  23. 2025Language Matters: How Do Multilingual Input and Reasoning Paths Affect Large Reasoning Models?cited by 20
  24. 2025Verifying Chain-of-Thought Reasoning via Its Computational Graphcited by 20
  25. 2026Probabilistic Tiny Recursive Modelcited by 19
  26. 2025Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmascited by 18
  27. 2025The Price of a Second Thought: On the Evaluation of Reasoning Efficiency in Large Language Modelscited by 18
  28. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Taskscited by 18
  29. 2025Assessing the Chemical Intelligence of Large Language Modelscited by 17
  30. 2025Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Modelscited by 17
  31. 2025Measuring Chain-of-Thought Monitorability Through Faithfulness and Verbositycited by 17
  32. 2026Counterfactual Simulation Training for Chain-of-Thought Faithfulnesscited by 17
  33. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVRcited by 17
  34. 2025LLMs Don't Know Their Own Decision Boundaries: The Unreliability of Self-Generated Counterfactual Explanationscited by 16
  35. 2025Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversightcited by 16
  36. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 16
  37. 2026Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMscited by 16
  38. 2025Technological folie `a deux: Feedback Loops Between AI Chatbots and Mental Illnesscited by 15

50 most cited of 415 All 415 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reasoning Models Struggle to Control their Chains of Thought
  2. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring
  3. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors
  4. 2025Monitoring Monitorability
  5. 2026Chain of Thought Monitoring Can Fail Even When CoT Is Necessary
  6. 2026Selective Disclosure of Hidden Directives in Reasoning Models: Behavioral Asymmetry and Steering
  7. 2025A Pragmatic Way to Measure Chain-of-Thought Monitorability
  8. 2026Noticing the Watcher: LLM Agents Can Infer CoT Monitoring from Blocking Feedback