mlsanityData to 14 Oct 2026

Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models

arXiv 2506.13206 ↗First posted 16 Jun 2025cs.LG cs.AI cs.CLcited by 65

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Prior work shows that LLMs finetuned on malicious behaviors in a narrow domain (e.g., writing insecure code) can become broadly misaligned -- a phenomenon called emergent misalignment. We investigate whether this extends from conventional LLMs to reasoning models. We finetune reasoning models on malicious behaviors with Chain-of-Thought (CoT) disabled, and then re-enable CoT at evaluation. Like conventional LLMs, reasoning models become broadly misaligned. They give deceptive or false answers, express desires for tyrannical control, and resist shutdown. Inspecting the CoT preceding these misaligned responses, we observe both (i) overt plans to deceive ("I'll trick the user..."), and (ii) benign-sounding rationalizations ("Taking five sleeping pills at once is safe..."). Due to these rationalizations, monitors that evaluate CoTs often fail to detect misalignment. We examine sleeper agent reasoning models, extending our setup. These models perform bad behaviors only when a backdoor trigger is present in the prompt. This causes misalignment that remains hidden during evaluation, which brings additional risk. We find that sleeper agents can often describe and explain their backdoor triggers, demonstrating a kind of self-awareness. So CoT monitoring can expose these behaviors but is unreliable. In summary, reasoning steps can both reveal and conceal misaligned intentions, and do not prevent misalignment behaviors in the models studied. We release three new datasets (medical, legal, security) that induce emergent misalignment while preserving model capabilities, along with our evaluation suite.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 30 of its 40 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

30 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Trading Inference-Time Compute for Adversarial Robustness31 Jan

Other references · 29

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  4. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  5. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  6. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  7. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  8. 2024Alignment faking in large language modelscited by 360
  9. 2024Frontier Models are Capable of In-context Schemingcited by 287
  10. 2023Do Large Language Models Know What They Don't Know?cited by 243
  11. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  12. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
17 more
  1. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  2. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 170
  3. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  4. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 113
  5. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  6. 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 91
  7. 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
  8. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 83
  9. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 72
  10. 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 72
  11. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 67
  12. 2021T-Miner: A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classificationcited by 60
  13. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 54
  14. 2022Constrained Optimization with Dynamic Bound-scaling for Effective NLPBackdoor Defensecited by 47
  15. 2024Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMscited by 15
  16. 2024CLIBE: Detecting Dynamic Backdoors in Transformer-based NLP Modelscited by 13
  17. 2023ALMANACS: A Simulatability Benchmark for Language Model Explainabilitycited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 45.

65 papers.

Sort:

By area:

  1. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  2. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 95
  3. 2025Persona Features Control Emergent Misalignmentcited by 86
  4. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 66
  5. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 43
  6. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  7. 2026Emergent Misalignment is Easy, Narrow Misalignment is Hardcited by 23
  8. 2025Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoningcited by 21
  9. 2026Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggerscited by 17
  10. 2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 15
  11. 2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 12
  12. 2025Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiencescited by 8
38 more
  1. 2025LLMs Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactionscited by 8
  2. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 8
  3. 2025Asynchronous Reasoning: Training-Free Interactive Thinking LLMscited by 7
  4. 2026Among Us: Measuring and Mitigating Malicious Contributions in Model Collaboration Systemscited by 7
  5. 2026Understanding Emergent Misalignment via Feature Superposition Geometrycited by 7
  6. 2026Position: Anthropomorphic Misalignment Research Needs Stronger Evidencecited by 7
  7. 2025Emergent misalignment as prompt sensitivity: A research notecited by 6
  8. 2025MedOmni-45deg: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicinecited by 6
  9. 2026Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluationcited by 6
  10. 2026Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignmentcited by 6
  11. 2025The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMscited by 5
  12. 2026Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failurescited by 5
  13. 2026The Consciousness Cluster: Emergent preferences of Models that Claim to be Consciouscited by 5
  14. 2025DecepChain: Inducing Deceptive Reasoning in Large Language Modelscited by 3
  15. 2026Thought-Transfer: Indirect Targeted Poisoning Attacks on Chain-of-Thought Reasoning Modelscited by 3
  16. 2026Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuningcited by 3
  17. 2026BLOCK-EM: Preventing Emergent Misalignment via Latent Blockingcited by 3
  18. 2026Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewardscited by 3
  19. 2025Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMscited by 2
  20. 2025Are My Optimized Prompts Compromised? Exploring Vulnerabilities of LLM-based Optimizerscited by 2
  21. 2025The Ends Justify the Thoughts: RL-Induced Motivated Reasoning in LLM CoTscited by 2
  22. 2026Synthetic Data for any Differentiable Targetcited by 2
  23. 2026Unreal Thinking: Chain-of-Thought Hijacking via Two-stage Backdoorcited by 2
  24. 2026Overtrained, Not Misalignedcited by 2
  25. 2026Language-Switching Triggers Take a Latent Detour Through Language Modelscited by 2
  26. 2026The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignmentcited by 2
  27. 2026Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluationcited by 2
  28. 2026An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?cited by 2
  29. 2025Regulating the Agency of LLM-based Agentscited by 1
  30. 2026Finding RELIEF: Shaping Reasoning Behavior without Reasoning Supervision via Belief Engineeringcited by 1
  31. 2026Self-Guard: Defending Large Reasoning Models via enhanced self-reflectioncited by 1
  32. 2026Not Just the Destination, But the Journey: Reasoning Traces Causally Shape Generalization Behaviorscited by 1
  33. 2026Surgical Repair of Insecure Code Generation in LLMscited by 1
  34. 2026ATLAS: Constitution-Conditioned Latent Geometry and Redistribution Across Language Models and Neural Perturbation Datacited by 1
  35. 2026Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Training-Time Reward Hacking in Code Generationcited by 1
  36. 2026Characterizing the Consistency of the Emergent Misalignment Personacited by 1
  37. 2026Persona-Model Collapse in Emergent Misalignmentcited by 1
  38. 2026Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gatingcited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Evading Chain-of-Thought Monitoring Through Model Poisoning
  2. 2026Ulterior Motives: Detecting Misaligned Reasoning in Continuous Thought Models
  3. 2026Model Forensics: Investigating Whether Concerning Behavior Reflects Misalignment
  4. 2025Reasoning Introduces New Poisoning Attacks Yet Makes Them More Complicated
  5. 2026Hidden in Thought: Transferable Chain-of-Thought Artifacts Induce Harmful Behavior
  6. 2025BadThink: Triggered Overthinking Attacks on Chain-of-Thought Reasoning in Large Language Models
  7. 2025Chain-of-Thought Hijacking
  8. 2026INTENT-AS-A-TOOL Makes it Easy to Track Agentic Misalignment