# Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models

arXiv: https://arxiv.org/abs/2506.13206 · HTML: https://mlsanity.com/paper/2506.13206/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 30)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [Multilingual Jailbreak Challenges in Large Language Models](https://mlsanity.com/paper/2310.06474/) (2023)
- [Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models](https://mlsanity.com/paper/2406.10162/) (2024)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Chain-of-Thought Reasoning In The Wild Is Not Always Faithful](https://mlsanity.com/paper/2503.08679/) (2025)
- [What is in Your Safe Data? Identifying Benign Data that Breaks Safety](https://mlsanity.com/paper/2404.01099/) (2024)
- [Looking Inward: Language Models Can Learn About Themselves by Introspection](https://mlsanity.com/paper/2410.13787/) (2024)
- [Faithfulness Tests for Natural Language Explanations](https://mlsanity.com/paper/2305.18029/) (2023)
- [Tell me about yourself: LLMs are aware of their learned behaviors](https://mlsanity.com/paper/2501.11120/) (2025)
- [Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations](https://mlsanity.com/paper/2307.08678/) (2023)
- [How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questions](https://mlsanity.com/paper/2309.15840/) (2023)
- [Are DeepSeek R1 And Other Reasoning Models More Faithful?](https://mlsanity.com/paper/2501.08156/) (2025)
- [The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems](https://mlsanity.com/paper/2503.03750/) (2025)
- [Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs](https://mlsanity.com/paper/2502.08640/) (2025)
- [Trading Inference-Time Compute for Adversarial Robustness](https://mlsanity.com/paper/2501.18841/) (2025)
- [T-Miner: A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classification](https://mlsanity.com/paper/2103.04264/) (2021)
- [CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring](https://mlsanity.com/paper/2505.23575/) (2025)
- [Constrained Optimization with Dynamic Bound-scaling for Effective NLPBackdoor Defense](https://mlsanity.com/paper/2202.05749/) (2022)
- [Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMs](https://mlsanity.com/paper/2407.04108/) (2024)
- [CLIBE: Detecting Dynamic Backdoors in Transformer-based NLP Models](https://mlsanity.com/paper/2409.01193/) (2024)
- [ALMANACS: A Simulatability Benchmark for Language Model Explainability](https://mlsanity.com/paper/2312.12747/) (2023)

## Built on by (most cited first, 30 of 65)

- [Natural Emergent Misalignment from Reward Hacking in Production RL](https://mlsanity.com/paper/2511.18397/) (2025)
- [Subliminal Learning: Language models transmit behavioral traits via hidden signals in data](https://mlsanity.com/paper/2507.14805/) (2025)
- [Persona Features Control Emergent Misalignment](https://mlsanity.com/paper/2506.19823/) (2025)
- [School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs](https://mlsanity.com/paper/2508.17511/) (2025)
- [Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time](https://mlsanity.com/paper/2510.04340/) (2025)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [Emergent Misalignment is Easy, Narrow Misalignment is Hard](https://mlsanity.com/paper/2602.07852/) (2026)
- [Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2506.22777/) (2025)
- [Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers](https://mlsanity.com/paper/2604.25891/) (2026)
- [In-Training Defenses against Emergent Misalignment in Language Models](https://mlsanity.com/paper/2508.06249/) (2025)
- [Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training](https://mlsanity.com/paper/2510.20956/) (2025)
- [Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiences](https://mlsanity.com/paper/2510.06105/) (2025)
- [LLMs Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactions](https://mlsanity.com/paper/2510.08211/) (2025)
- [Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer](https://mlsanity.com/paper/2605.12798/) (2026)
- [Asynchronous Reasoning: Training-Free Interactive Thinking LLMs](https://mlsanity.com/paper/2512.10931/) (2025)
- [Among Us: Measuring and Mitigating Malicious Contributions in Model Collaboration Systems](https://mlsanity.com/paper/2602.05176/) (2026)
- [Understanding Emergent Misalignment via Feature Superposition Geometry](https://mlsanity.com/paper/2605.00842/) (2026)
- [Position: Anthropomorphic Misalignment Research Needs Stronger Evidence](https://mlsanity.com/paper/2606.07612/) (2026)
- [Emergent misalignment as prompt sensitivity: A research note](https://mlsanity.com/paper/2507.06253/) (2025)
- [MedOmni-45deg: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine](https://mlsanity.com/paper/2508.16213/) (2025)
- [Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation](https://mlsanity.com/paper/2601.14691/) (2026)
- [Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignment](https://mlsanity.com/paper/2602.14777/) (2026)
- [The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMs](https://mlsanity.com/paper/2511.20104/) (2025)
- [Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures](https://mlsanity.com/paper/2601.23081/) (2026)
- [The Consciousness Cluster: Emergent preferences of Models that Claim to be Conscious](https://mlsanity.com/paper/2604.13051/) (2026)
- [DecepChain: Inducing Deceptive Reasoning in Large Language Models](https://mlsanity.com/paper/2510.00319/) (2025)
- [Thought-Transfer: Indirect Targeted Poisoning Attacks on Chain-of-Thought Reasoning Models](https://mlsanity.com/paper/2601.19061/) (2026)
- [Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuning](https://mlsanity.com/paper/2602.00298/) (2026)
- [BLOCK-EM: Preventing Emergent Misalignment via Latent Blocking](https://mlsanity.com/paper/2602.00767/) (2026)
- [Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards](https://mlsanity.com/paper/2605.31328/) (2026)

## Abstract (arXiv)

Prior work shows that LLMs finetuned on malicious behaviors in a narrow domain (e.g., writing insecure code) can become broadly misaligned -- a phenomenon called emergent misalignment. We investigate whether this extends from conventional LLMs to reasoning models. We finetune reasoning models on malicious behaviors with Chain-of-Thought (CoT) disabled, and then re-enable CoT at evaluation. Like conventional LLMs, reasoning models become broadly misaligned. They give deceptive or false answers, express desires for tyrannical control, and resist shutdown. Inspecting the CoT preceding these misaligned responses, we observe both (i) overt plans to deceive ("I'll trick the user..."), and (ii) benign-sounding rationalizations ("Taking five sleeping pills at once is safe..."). Due to these rationalizations, monitors that evaluate CoTs often fail to detect misalignment. We examine sleeper agent reasoning models, extending our setup. These models perform bad behaviors only when a backdoor trigger is present in the prompt. This causes misalignment that remains hidden during evaluation, which brings additional risk. We find that sleeper agents can often describe and explain their backdoor triggers, demonstrating a kind of self-awareness. So CoT monitoring can expose these behaviors but is unreliable. In summary, reasoning steps can both reveal and conceal misaligned intentions, and do not prevent misalignment behaviors in the models studied. We release three new datasets (medical, legal, security) that induce emergent misalignment while preserving model capabilities, along with our evaluation suite.