mlsanityData to 14 Oct 2026

Convergent Linear Representations of Emergent Misalignment

arXiv 2506.11618 ↗First posted 13 Jun 2025cs.LG cs.AIcited by 71

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large language models on narrow datasets can cause them to develop broadly misaligned behaviours: a phenomena known as emergent misalignment. However, the mechanisms underlying this misalignment, and why it generalizes beyond the training domain, are poorly understood, demonstrating critical gaps in our knowledge of model alignment. In this work, we train and study a minimal model organism which uses just 9 rank-1 adapters to emergently misalign Qwen2.5-14B-Instruct. Studying this, we find that different emergently misaligned models converge to similar representations of misalignment. We demonstrate this convergence by extracting a 'misalignment direction' from one fine-tuned model's activations, and using it to effectively ablate misaligned behaviour from fine-tunes using higher dimensional LoRAs and different datasets. Leveraging the scalar hidden state of rank-1 LoRAs, we further present a set of experiments for directly interpreting the fine-tuning adapters, showing that six contribute to general misalignment, while two specialise for misalignment in just the fine-tuning domain. Emergent misalignment is a particularly salient example of undesirable and unexpected model behaviour and by advancing our understanding of the mechanisms behind it, we hope to move towards being able to better understand and mitigate misalignment more generally.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Builds on 18 of its 32 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

18 of 32 references.

References · 18

  1. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  2. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  4. 2023Steering Language Models With Activation Engineeringcited by 922
  5. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  6. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  7. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  8. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  9. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  10. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  11. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  12. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
6 more
  1. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 115
  2. 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 91
  3. 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 84
  4. 2022Gold Doesn't Always Glitter: Spectral Removal of Linear and Nonlinear Guarded Attribute Informationcited by 34
  5. 2022Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projectioncited by 17
  6. 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 60.

71 papers.

Sort:

By area:

  1. 2025Persona Features Control Emergent Misalignmentcited by 86
  2. 2026Emergent Misalignment is Easy, Narrow Misalignment is Hardcited by 23
  3. 2025Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMscited by 17
  4. 2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 15
  5. 2026Monitoring Emergent Reward Hacking During Generation via Internal Activationscited by 14
  6. 2026Subliminal Learning Is Steering Vector Distillationcited by 13
  7. 2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 12
  8. 2026Where is the Mind? Persona Vectors and LLM Individuationcited by 10
  9. 2026LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuitcited by 10
  10. 2025Shared Parameter Subspaces and Cross-Task Linearity in Emergently Misaligned Behaviorcited by 8
  11. 2026Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attributioncited by 8
  12. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 8
38 more
  1. 2026Mechanisms of Introspective Awarenesscited by 7
  2. 2026Understanding Emergent Misalignment via Feature Superposition Geometrycited by 7
  3. 2026Position: Anthropomorphic Misalignment Research Needs Stronger Evidencecited by 7
  4. 2025Emergent misalignment as prompt sensitivity: A research notecited by 6
  5. 2026Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignmentcited by 6
  6. 2026Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agentscited by 6
  7. 2025Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignmentcited by 5
  8. 2025The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMscited by 5
  9. 2026Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failurescited by 5
  10. 2025Memory Retrieval and Consolidation in Large Language Models through Function Tokenscited by 4
  11. 2026Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimescited by 4
  12. 2026From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretabilitycited by 4
  13. 2026Activation Steering for Aligned Open-ended Generation without Sacrificing Coherencecited by 4
  14. 2025Latent Concept Disentanglement in Transformer-based Language Modelscited by 3
  15. 2026Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuningcited by 3
  16. 2026From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agentscited by 3
  17. 2026Emergent Misalignment Recruits a Pre-existing Persona Subspacecited by 3
  18. 2026The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignmentcited by 2
  19. 2026Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluationcited by 2
  20. 2026An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?cited by 2
  21. 2025The Horcrux: Mechanistically Interpretable Task Decomposition for Detecting and Mitigating Reward Hacking in Embodied AI Systemscited by 1
  22. 2026The Shadow Self: Intrinsic Value Misalignment in Large Language Model Agentscited by 1
  23. 2026A Geometric Taxonomy of Hallucinations in LLMscited by 1
  24. 2026Value Entanglement: Conflation Between Different Kinds of Good In (Some) Large Language Modelscited by 1
  25. 2026Not Just the Destination, But the Journey: Reasoning Traces Causally Shape Generalization Behaviorscited by 1
  26. 2026Why Safety Probes Catch Liars But Miss Fanaticscited by 1
  27. 2026Weird Generalization is Weirdly Brittlecited by 1
  28. 2026Surgical Repair of Insecure Code Generation in LLMscited by 1
  29. 2026Characterizing the Consistency of the Emergent Misalignment Personacited by 1
  30. 2026Persona-Model Collapse in Emergent Misalignmentcited by 1
  31. 2026Unsupervised Identification and Removal of Spurious Correlations During Fine-Tuningcited by 1
  32. 2026Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gatingcited by 1
  33. 2026Actionable Activation Directions for Detecting and Mitigating Emergent Misalignment Across Language Model Familiescited by 1
  34. 2026Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignmentcited by 1
  35. 2026Revealing Hidden Model Behaviors with Task-Specific Self-Reportscited by 1
  36. 2026Data Attribution of Emergent Misalignment with Persona Featurescited by 1
  37. 2026TAME: Token Attribution and Masking for Emergent misalignmentcited by 1
  38. 2026Measuring the Assistant's Harmlessness Preferences on the User Turncited by 1

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Model Organisms for Emergent Misalignment
  2. 2026Overtrained, Not Misaligned
  3. 2026Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards
  4. 2026Are Aligned Large Language Models Still Misaligned?
  5. 2025Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMs
  6. 2026Ventriloquist LLMs: Linear Alignment of Late-Stage Representations
  7. 2024Language Models Resist Alignment: Evidence From Data Compression
  8. 2026Alignment Dynamics in LLM Fine-Tuning