# Emergent Misalignment is Easy, Narrow Misalignment is Hard

arXiv: https://arxiv.org/abs/2602.07852 · HTML: https://mlsanity.com/paper/2602.07852/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 8 Feb 2026.

## Builds on (most cited first, 23 of 23)

- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Sharp Minima Can Generalize For Deep Nets](https://mlsanity.com/paper/1703.04933/) (2017)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [Gradient Descent Maximizes the Margin of Homogeneous Neural Networks](https://mlsanity.com/paper/1906.05890/) (2019)
- [Taken out of context: On measuring situational awareness in LLMs](https://mlsanity.com/paper/2309.00667/) (2023)
- [A Bayesian Perspective on Generalization and Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06451/) (2017)
- [Explaining grokking through circuit efficiency](https://mlsanity.com/paper/2309.02390/) (2023)
- [Tell me about yourself: LLMs are aware of their learned behaviors](https://mlsanity.com/paper/2501.11120/) (2025)
- [Model Organisms for Emergent Misalignment](https://mlsanity.com/paper/2506.11613/) (2025)
- [Convergent Linear Representations of Emergent Misalignment](https://mlsanity.com/paper/2506.11618/) (2025)
- [Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models](https://mlsanity.com/paper/2506.13206/) (2025)
- [The alignment property of SGD noise and how it helps select flat minima: A stability analysis](https://mlsanity.com/paper/2207.02628/) (2022)
- [The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networks](https://mlsanity.com/paper/2012.06244/) (2020)
- [Compromising Honesty and Harmlessness in Language Models via Deception Attacks](https://mlsanity.com/paper/2502.08301/) (2025)

## Built on by (most cited first, 23 of 23)

- [Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers](https://mlsanity.com/paper/2604.25891/) (2026)
- [Subliminal Steering: Stronger Encoding of Hidden Signals](https://mlsanity.com/paper/2604.25783/) (2026)
- [Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer](https://mlsanity.com/paper/2605.12798/) (2026)
- [Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types](https://mlsanity.com/paper/2604.09544/) (2026)
- [Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards](https://mlsanity.com/paper/2605.31328/) (2026)
- [Emergent Misalignment Recruits a Pre-existing Persona Subspace](https://mlsanity.com/paper/2607.21356/) (2026)
- [The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment](https://mlsanity.com/paper/2606.06667/) (2026)
- [Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluation](https://mlsanity.com/paper/2606.08682/) (2026)
- [Filtering Harmful Actions Isn't Enough: Phantom Transfer in Agentic SDF](https://mlsanity.com/paper/2607.10750/) (2026)
- [Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating](https://mlsanity.com/paper/2606.09068/) (2026)
- [Epistemic Goggles: A Pretrained Module that Induces an Epistemic Frame via Gradient Editing](https://mlsanity.com/paper/2607.01690/) (2026)
- [Innocuous-Seeming Data, Latent Ideology: Ideological Generalisation in Finetuned LLMs](https://mlsanity.com/paper/2607.14888/) (2026)
- [Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models](https://mlsanity.com/paper/2607.26173/) (2026)
- [The Unequal Influence of Bad Advice: Using Training Data Attribution to Modulate Emergent Misalignment](https://mlsanity.com/paper/2609.37914/) (2026)
- [Phase Transitions in Driven Informational Systems: A Two-Field Perspective on Learning Theory and Non-Equilibrium Chemistry](https://mlsanity.com/paper/2605.16325/) (2026)
- [What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data](https://mlsanity.com/paper/2606.20814/) (2026)
- [Evil Spectra: How Optimisers can Amplify or Suppress Emergent Misalignment](https://mlsanity.com/paper/2606.31591/) (2026)
- [Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.5](https://mlsanity.com/paper/2607.04510/) (2026)
- [Emergent Misalignment Is Not Magical](https://mlsanity.com/paper/2608.29118/) (2026)
- [Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Text](https://mlsanity.com/paper/2608.30619/) (2026)
- [When Do Models Admit They Are Wrong? Failure Disclosure Is Unstable Under Reinforcement Learning](https://mlsanity.com/paper/2609.33220/) (2026)
- [Correct, Don't Delete: Mitigating Emergent Misalignment with Corrective Supervision](https://mlsanity.com/paper/2609.37624/) (2026)
- [Aligned Data Can Induce Misalignment via Context Confusion](https://mlsanity.com/paper/2609.38379/) (2026)

## Abstract (arXiv)

Finetuning large language models on narrowly harmful datasets can cause them to become emergently misaligned, giving stereotypically `evil' responses across diverse unrelated settings. Concerningly, a pre-registered survey of experts failed to predict this result, highlighting our poor understanding of the inductive biases governing learning and generalisation in LLMs. We use emergent misalignment (EM) as a case study to investigate these inductive biases and find that models can just learn the narrow dataset task, but that the general solution appears to be more stable and more efficient. To establish this, we build on the result that different EM finetunes converge to the same linear representation of general misalignment, which can be used to mediate misaligned behaviour. We find a linear representation of the narrow solution also exists, and can be learned by introducing a KL divergence loss. Comparing these representations reveals that general misalignment achieves lower loss, is more robust to perturbations, and is more influential in the pre-training distribution. This work isolates a concrete representation of general misalignment for monitoring and mitigation. More broadly, it offers a detailed case study and preliminary metrics for investigating how inductive biases shape generalisation in LLMs. We open-source all code, datasets and model finetunes.