mlsanityData to 7 Oct 2026

Training Agents to Self-Report Misbehavior

arXiv 2602.22303 ↗First posted 25 Feb 2026cs.LG cs.AIcited by 3

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Frontier AI agents may pursue hidden goals while concealing their pursuit from oversight. Alignment training aims to prevent such behavior by reinforcing the correct goals, but alignment may not always succeed and can lead to unwanted side effects. We propose self-incrimination training, which instead trains agents to produce a visible signal when they covertly misbehave. We train GPT-4.1 and Gemini-2.0 agents to call a report_scheming() tool when behaving deceptively and measure their ability to cause harm undetected in out-of-distribution environments. Self-incrimination significantly reduces the undetected successful attack rate, outperforming matched-capability monitors and alignment baselines while preserving instruction hierarchy and incurring minimal safety tax on general capabilities. Unlike blackbox monitoring, self-incrimination performance is consistent across tasks regardless of how suspicious the misbehavior appears externally. The trained behavior persists under adversarial prompt optimization and generalizes to settings where agents pursue misaligned goals themselves rather than being instructed to misbehave. Our results suggest self-incrimination offers a viable path for reducing frontier misalignment risk, one that neither assumes misbehavior can be prevented nor that it can be reliably classified from the outside.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 89 of its 112 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

89 of 112 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025BashArena: A Control Setting for Highly Privileged AI Agents17 Dec
  2. 2025Async Control: Stress-testing Asynchronous Control Measures for LLM Agents15 Dec
  3. 2025Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols10 Oct
  4. 2025Agentic Misalignment: How LLMs Could Be Insider Threats5 Oct
  5. 2025Reliable Weak-to-Strong Monitoring of LLM Agents26 Aug
  6. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents17 Jun
  7. 2025Control Tax: The Price of Keeping AI in Check5 Jun
  8. 2025A sketch of an AI control safety case28 Jan
  9. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents11 Oct
  10. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
  11. 2023Testing Language Model Agents Safely in the Wild17 Nov

Other references · 78

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  9. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2021Measuring Coding Challenge Competence With APPScited by 879
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
38 more
  1. 2023Steering Language Models With Activation Engineeringcited by 824
  2. 2020Aligning AI With Shared Human Valuescited by 706
  3. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  4. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  5. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  6. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  7. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  8. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  9. 2024Alignment faking in large language modelscited by 316
  10. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  11. 2024Frontier Models are Capable of In-context Schemingcited by 252
  12. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  13. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  14. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  15. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 222
  16. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  17. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  18. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  19. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  20. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 172
  21. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 166
  22. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  23. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  24. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 120
  25. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  26. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  27. 2024Are self-explanations from Large Language Models faithful?cited by 101
  28. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  29. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 90
  30. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 81
  31. 2024Visibility into AI Agentscited by 77
  32. 2025Persona Features Control Emergent Misalignmentcited by 76
  33. 2025Stress Testing Deliberative Alignment for Anti-Scheming Trainingcited by 71
  34. 2024Learning to Route LLMs with Confidence Tokenscited by 66
  35. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 66
  36. 2025Detecting Strategic Deception Using Linear Probescited by 65
  37. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 49
  38. 2025From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Trainingcited by 49

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Decoding Hidden Deception in Reasoning LLMs: Activation Explainers for Deception Auditingcited by 1
  2. 2026Can LLMs Reliably Self-Report Adversarial Prefills, and How?cited by 0
  3. 2026When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Training Deliberative Monitors for Black-Box Scheming Detection
  2. 2026Why Safety Probes Catch Liars But Miss Fanatics
  3. 2026The Autonomy Tax: Defense Training Breaks LLM Agents
  4. 2026Self-Propagating Misalignment in LLM Agents, and Why Auditing or Disabling Memory Is Not Enough
  5. 2026Aligned Alone, Misaligned Together: Forecasting Adversarial Capture in LLM Agent Populations
  6. 2026Self-Reflection Fine-Tuning: Enhancing Agent Security against Prompt Injection Attacks from Failure Experience
  7. 2026Self-Attribution Bias: When AI Monitors Go Easy on Themselves
  8. 2024Towards Safe and Honest AI Agents with Neural Self-Other Overlap