mlsanityData to 7 Oct 2026

Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation

arXiv 2503.11926 ↗First posted 14 Mar 2025cs.AIcited by 338

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

Mitigating reward hacking--where AI systems misbehave due to flaws or misspecifications in their learning objectives--remains a key challenge in constructing capable and aligned models. We show that we can monitor a frontier reasoning model, such as OpenAI o3-mini, for reward hacking in agentic coding environments by using another LLM that observes the model's chain-of-thought (CoT) reasoning. CoT monitoring can be far more effective than monitoring agent actions and outputs alone, and we further found that a LLM weaker than o3-mini, namely GPT-4o, can effectively monitor a stronger model. Because CoT monitors can be effective at detecting exploits, it is natural to ask whether those exploits can be suppressed by incorporating a CoT monitor directly into the agent's training objective. While we show that integrating CoT monitors into the reinforcement learning reward can indeed produce more capable and more aligned agents in the low optimization regime, we find that with too much optimization, agents learn obfuscated reward hacking, hiding their intent within the CoT while still exhibiting a significant rate of reward hacking. Because it is difficult to tell when CoTs have become obfuscated, it may be necessary to pay a monitorability tax by not applying strong optimization pressures directly to the chain-of-thought, ensuring that CoTs remain monitorable and useful for detecting misaligned behavior.

Key facts. Cited by 338 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 48 of its 72 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

48 of 72 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step23 May
  2. 2023On Measuring Faithfulness or Self-consistency of Natural Language Explanations13 Nov
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks22 Nov
  4. 2022Large Language Models are Zero-Shot Reasoners24 May
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
30 more
  1. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  5. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  6. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  7. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  8. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  9. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  10. 2022PAL: Program-aided Language Modelscited by 635
  11. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  12. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  13. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  14. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  15. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  16. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  17. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  18. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  19. 2024Frontier Models are Capable of In-context Schemingcited by 252
  20. 2021Internet-Augmented Dialogue Generationcited by 217
  21. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  22. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  23. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  24. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  25. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  26. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  27. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  28. 2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 61
  29. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 56
  30. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 124; 2026: 214.

338 papers.

Sort:

By area:

  1. 2025Reasoning Models Don't Always Say What They Thinkcited by 388
  2. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 222
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  5. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 126
  6. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  7. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  8. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
  9. 2025Persona Features Control Emergent Misalignmentcited by 76
  10. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 63
  11. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agentscited by 58
  12. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 58
38 more
  1. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 57
  2. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 49
  3. 2025Detecting High-Stakes Interactions with Activation Probescited by 45
  4. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  5. 2025Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokenscited by 41
  6. 2025Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activationscited by 37
  7. 2025Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!cited by 36
  8. 2026Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thoughtcited by 35
  9. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  10. 2025Large language models can learn and generalize steganographic chain-of-thought under process supervisioncited by 30
  11. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  12. 2025The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awarenesscited by 29
  13. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  14. 2025Security Concerns for Large Language Models: A Surveycited by 26
  15. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 25
  16. 2025Leaky Thoughts: Large Reasoning Models Are Not Private Thinkerscited by 25
  17. 2025Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effortcited by 25
  18. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agentscited by 25
  19. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 24
  20. 2025EvilGenie: A Reward Hacking Benchmarkcited by 24
  21. 2026LLMs Gaming Verifiers: RLVR can Lead to Reward Hackingcited by 24
  22. 2025Self-Interpretability: LLMs Can Describe Complex Internal Processes that Drive Their Decisionscited by 23
  23. 2025Measuring the Faithfulness of Thinking Drafts in Large Reasoning Modelscited by 22
  24. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  25. 2026Reasoning Models Struggle to Control their Chains of Thoughtcited by 22
  26. 2025How to evaluate control measures for LLM agents? A trajectory from today to superintelligencecited by 20
  27. 2025Verifying Chain-of-Thought Reasoning via Its Computational Graphcited by 20
  28. 2026Large Language Model Reasoning Failurescited by 20
  29. 2025From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoningcited by 19
  30. 2025RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?cited by 19
  31. 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 19
  32. 2025Training LLMs for Honesty via Confessionscited by 19
  33. 2026SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?cited by 19
  34. 2025Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorabilitycited by 18
  35. 2025Async Control: Stress-testing Asynchronous Control Measures for LLM Agentscited by 18
  36. 2025Benchmarking Deception Probes via Black-to-White Performance Boostscited by 17
  37. 2025LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoringcited by 17
  38. 2025IRIS: Interactive Research Ideation System for Accelerating Scientific Discoverycited by 16

50 most cited of 338 All 338 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Training on Documents About Monitoring Leads to CoT Obfuscation
  2. 2025Monitoring Monitorability
  3. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning
  4. 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering
  5. 2026Evading Chain-of-Thought Monitoring Through Model Poisoning
  6. 2026hacktrace: behavior-supervised detection of reward hacking during code generation
  7. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
  8. 2026BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks