# Model Organisms for Emergent Misalignment arXiv: https://arxiv.org/abs/2506.11613 ยท HTML: https://mlsanity.com/paper/2506.11613/ ## Key facts - Cited by 91 papers in the map (data through 14 Oct 2026). - Builds on 21 of its 33 references that are in the map. - First posted on arXiv 13 Jun 2025. ## Builds on (most cited first, 21 of 21) - [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022) - [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024) - [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018) - [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023) - [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023) - [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022) - [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024) - [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023) - [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023) - [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023) - [Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models](https://mlsanity.com/paper/2405.16833/) (2024) - [General-Purpose In-Context Learning by Meta-Learning Transformers](https://mlsanity.com/paper/2212.04458/) (2022) - [Tell me about yourself: LLMs are aware of their learned behaviors](https://mlsanity.com/paper/2501.11120/) (2025) - [Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs](https://mlsanity.com/paper/2407.04694/) (2024) - [Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization](https://mlsanity.com/paper/2405.15071/) (2024) - [SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation](https://mlsanity.com/paper/2501.01765/) (2025) - [Loss Landscape Degeneracy and Stagewise Development in Transformers](https://mlsanity.com/paper/2402.02364/) (2024) - [Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space](https://mlsanity.com/paper/2406.19370/) (2024) - [A Geometric Notion of Causal Probing](https://mlsanity.com/paper/2307.15054/) (2023) - [Compromising Honesty and Harmlessness in Language Models via Deception Attacks](https://mlsanity.com/paper/2502.08301/) (2025) - [Phase Transitions in the Output Distribution of Large Language Models](https://mlsanity.com/paper/2405.17088/) (2024) ## Built on by (most cited first, 30 of 91) - [Natural Emergent Misalignment from Reward Hacking in Production RL](https://mlsanity.com/paper/2511.18397/) (2025) - [Subliminal Learning: Language models transmit behavioral traits via hidden signals in data](https://mlsanity.com/paper/2507.14805/) (2025) - [Persona Features Control Emergent Misalignment](https://mlsanity.com/paper/2506.19823/) (2025) - [School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs](https://mlsanity.com/paper/2508.17511/) (2025) - [Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainers](https://mlsanity.com/paper/2512.15674/) (2025) - [Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning](https://mlsanity.com/paper/2507.16795/) (2025) - [Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer](https://mlsanity.com/paper/2509.23886/) (2025) - [Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences](https://mlsanity.com/paper/2510.13900/) (2025) - [Emergent Misalignment is Easy, Narrow Misalignment is Hard](https://mlsanity.com/paper/2602.07852/) (2026) - [Steering Language Models with Weight Arithmetic](https://mlsanity.com/paper/2511.05408/) (2025) - [Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers](https://mlsanity.com/paper/2604.25891/) (2026) - [In-Training Defenses against Emergent Misalignment in Language Models](https://mlsanity.com/paper/2508.06249/) (2025) - [Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training](https://mlsanity.com/paper/2510.20956/) (2025) - [Where is the Mind? Persona Vectors and LLM Individuation](https://mlsanity.com/paper/2604.17031/) (2026) - [Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiences](https://mlsanity.com/paper/2510.06105/) (2025) - [LLMs Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactions](https://mlsanity.com/paper/2510.08211/) (2025) - [Shared Parameter Subspaces and Cross-Task Linearity in Emergently Misaligned Behavior](https://mlsanity.com/paper/2511.02022/) (2025) - [Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution](https://mlsanity.com/paper/2602.14869/) (2026) - [Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer](https://mlsanity.com/paper/2605.12798/) (2026) - [Position: Anthropomorphic Misalignment Research Needs Stronger Evidence](https://mlsanity.com/paper/2606.07612/) (2026) - [Emergent misalignment as prompt sensitivity: A research note](https://mlsanity.com/paper/2507.06253/) (2025) - [When Thinking Backfires: Mechanistic Insights Into Reasoning-Induced Misalignment](https://mlsanity.com/paper/2509.00544/) (2025) - [Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models](https://mlsanity.com/paper/2601.15220/) (2026) - [Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignment](https://mlsanity.com/paper/2602.14777/) (2026) - [Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents](https://mlsanity.com/paper/2602.16931/) (2026) - [Subliminal Learning is a LoRA Artifact](https://mlsanity.com/paper/2606.00831/) (2026) - [Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignment](https://mlsanity.com/paper/2508.20015/) (2025) - [The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMs](https://mlsanity.com/paper/2511.20104/) (2025) - [The Consciousness Cluster: Emergent preferences of Models that Claim to be Conscious](https://mlsanity.com/paper/2604.13051/) (2026) - [Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimes](https://mlsanity.com/paper/2603.04426/) (2026) ## Abstract (arXiv) Recent work discovered Emergent Misalignment (EM): fine-tuning large language models on narrowly harmful datasets can lead them to become broadly misaligned. A survey of experts prior to publication revealed this was highly unexpected, demonstrating critical gaps in our understanding of model alignment. In this work, we both advance understanding and provide tools for future research. Using new narrowly misaligned datasets, we create a set of improved model organisms that achieve 99% coherence (vs. 67% prior), work with smaller 0.5B parameter models (vs. 32B), and that induce misalignment using a single rank-1 LoRA adapter. We demonstrate that EM occurs robustly across diverse model sizes, three model families, and numerous training protocols including full supervised fine-tuning. Leveraging these cleaner model organisms, we isolate a mechanistic phase transition and demonstrate that it corresponds to a robust behavioural phase transition in all studied organisms. Aligning large language models is critical for frontier AI safety, yet EM exposes how far we are from achieving this robustly. By distilling clean model organisms that isolate a minimal alignment-compromising change, and where this is learnt, we establish a foundation for future research into understanding and mitigating alignment risks in LLMs.