Semantic Containment as a Fundamental Property of Emergent Misalignment
Not on the map. Why?
Abstract (arXiv)
Fine-tuning language models on narrowly harmful data causes emergent misalignment (EM) -- behavioral failures extending far beyond training distributions. Recent work demonstrates compartmentalization of misalignment behind contextual triggers, but these experiments mixed 97% benign data with 3% harmful triggered data. We investigate whether this mix of benign and harmful data teaches models to compartmentalize, or whether semantic triggers alone create containment. We train three model families (Qwen 2.5 14B, Llama 3.1 8B, Gemma 3 12B) with zero benign data -- only harmful examples with triggers, eliminating the good-bad data contrast. We demonstrate that baseline EM rates of 9.5--23.5% drop to 0.0--1.0% when triggers are removed during inference, but recover to 12.2--22.8% when triggers are present -- despite never seeing benign behavior to contrast against. Rephrased triggers maintain this containment, revealing that models respond to semantic meaning rather than surface syntax. These results show that semantic triggers spontaneously induce compartmentalization without requiring a mix of benign and harmful training data, exposing a critical safety gap: any harmful fine-tuning with contextual framing creates exploitable vulnerabilities invisible to standard evaluation.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 2 Feb 2026.
Builds on
26 of 36 references.
References · 26
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
- 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
- 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
- 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
- 2018Detecting Backdoor Attacks on Deep Neural Networks by Activation Clusteringcited by 429
- 2020Backdoor Learning: A Surveycited by 412
- 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
- 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
- 2023Poisoning Language Models During Instruction Tuningcited by 254
14 more
- 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
- 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
- 2020Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defensescited by 171
- 2023Linear Representations of Sentiment in Large Language Modelscited by 157
- 2020Dynamic Backdoor Attacks Against Machine Learning Modelscited by 147
- 2023Taken out of context: On measuring situational awareness in LLMscited by 131
- 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
- 2023On the Exploitability of Instruction Tuningcited by 110
- 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
- 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 86
- 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 78
- 2025Convergent Linear Representations of Emergent Misalignmentcited by 66
- 2025Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMscited by 47
- 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
Built on by
Not cited yet.