# Convergent Linear Representations of Emergent Misalignment

arXiv: https://arxiv.org/abs/2506.11618 · HTML: https://mlsanity.com/paper/2506.11618/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Builds on (most cited first, 18 of 18)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023)
- [The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence](https://mlsanity.com/paper/2502.17420/) (2025)
- [Tell me about yourself: LLMs are aware of their learned behaviors](https://mlsanity.com/paper/2501.11120/) (2025)
- [Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs](https://mlsanity.com/paper/2407.04694/) (2024)
- [Gold Doesn't Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Information](https://mlsanity.com/paper/2203.07893/) (2022)
- [Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection](https://mlsanity.com/paper/2212.04273/) (2022)
- [Compromising Honesty and Harmlessness in Language Models via Deception Attacks](https://mlsanity.com/paper/2502.08301/) (2025)

## Built on by (most cited first, 30 of 71)

- [Persona Features Control Emergent Misalignment](https://mlsanity.com/paper/2506.19823/) (2025)
- [Emergent Misalignment is Easy, Narrow Misalignment is Hard](https://mlsanity.com/paper/2602.07852/) (2026)
- [Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs](https://mlsanity.com/paper/2510.11288/) (2025)
- [In-Training Defenses against Emergent Misalignment in Language Models](https://mlsanity.com/paper/2508.06249/) (2025)
- [Monitoring Emergent Reward Hacking During Generation via Internal Activations](https://mlsanity.com/paper/2603.04069/) (2026)
- [Subliminal Learning Is Steering Vector Distillation](https://mlsanity.com/paper/2606.00995/) (2026)
- [Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training](https://mlsanity.com/paper/2510.20956/) (2025)
- [Where is the Mind? Persona Vectors and LLM Individuation](https://mlsanity.com/paper/2604.17031/) (2026)
- [LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuit](https://mlsanity.com/paper/2604.19117/) (2026)
- [Shared Parameter Subspaces and Cross-Task Linearity in Emergently Misaligned Behavior](https://mlsanity.com/paper/2511.02022/) (2025)
- [Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attribution](https://mlsanity.com/paper/2602.14869/) (2026)
- [Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer](https://mlsanity.com/paper/2605.12798/) (2026)
- [Mechanisms of Introspective Awareness](https://mlsanity.com/paper/2603.21396/) (2026)
- [Understanding Emergent Misalignment via Feature Superposition Geometry](https://mlsanity.com/paper/2605.00842/) (2026)
- [Position: Anthropomorphic Misalignment Research Needs Stronger Evidence](https://mlsanity.com/paper/2606.07612/) (2026)
- [Emergent misalignment as prompt sensitivity: A research note](https://mlsanity.com/paper/2507.06253/) (2025)
- [Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignment](https://mlsanity.com/paper/2602.14777/) (2026)
- [Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agents](https://mlsanity.com/paper/2602.16931/) (2026)
- [Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignment](https://mlsanity.com/paper/2508.20015/) (2025)
- [The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMs](https://mlsanity.com/paper/2511.20104/) (2025)
- [Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures](https://mlsanity.com/paper/2601.23081/) (2026)
- [Memory Retrieval and Consolidation in Large Language Models through Function Tokens](https://mlsanity.com/paper/2510.08203/) (2025)
- [Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimes](https://mlsanity.com/paper/2603.04426/) (2026)
- [From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretability](https://mlsanity.com/paper/2603.29805/) (2026)
- [Activation Steering for Aligned Open-ended Generation without Sacrificing Coherence](https://mlsanity.com/paper/2604.08169/) (2026)
- [Latent Concept Disentanglement in Transformer-based Language Models](https://mlsanity.com/paper/2506.16975/) (2025)
- [Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuning](https://mlsanity.com/paper/2602.00298/) (2026)
- [From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents](https://mlsanity.com/paper/2606.06223/) (2026)
- [Emergent Misalignment Recruits a Pre-existing Persona Subspace](https://mlsanity.com/paper/2607.21356/) (2026)
- [The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment](https://mlsanity.com/paper/2606.06667/) (2026)

## Abstract (arXiv)

Fine-tuning large language models on narrow datasets can cause them to develop broadly misaligned behaviours: a phenomena known as emergent misalignment. However, the mechanisms underlying this misalignment, and why it generalizes beyond the training domain, are poorly understood, demonstrating critical gaps in our knowledge of model alignment. In this work, we train and study a minimal model organism which uses just 9 rank-1 adapters to emergently misalign Qwen2.5-14B-Instruct. Studying this, we find that different emergently misaligned models converge to similar representations of misalignment. We demonstrate this convergence by extracting a 'misalignment direction' from one fine-tuned model's activations, and using it to effectively ablate misaligned behaviour from fine-tunes using higher dimensional LoRAs and different datasets. Leveraging the scalar hidden state of rank-1 LoRAs, we further present a set of experiments for directly interpreting the fine-tuning adapters, showing that six contribute to general misalignment, while two specialise for misalignment in just the fine-tuning domain. Emergent misalignment is a particularly salient example of undesirable and unexpected model behaviour and by advancing our understanding of the mechanisms behind it, we hope to move towards being able to better understand and mitigate misalignment more generally.