mlsanityData to 14 Oct 2026

Model Organisms for Emergent Misalignment

arXiv 2506.11613 ↗First posted 13 Jun 2025cs.LG cs.AIcited by 91

Not on the map. Why?

Abstract (arXiv)

Recent work discovered Emergent Misalignment (EM): fine-tuning large language models on narrowly harmful datasets can lead them to become broadly misaligned. A survey of experts prior to publication revealed this was highly unexpected, demonstrating critical gaps in our understanding of model alignment. In this work, we both advance understanding and provide tools for future research. Using new narrowly misaligned datasets, we create a set of improved model organisms that achieve 99% coherence (vs. 67% prior), work with smaller 0.5B parameter models (vs. 32B), and that induce misalignment using a single rank-1 LoRA adapter. We demonstrate that EM occurs robustly across diverse model sizes, three model families, and numerous training protocols including full supervised fine-tuning. Leveraging these cleaner model organisms, we isolate a mechanistic phase transition and demonstrate that it corresponds to a robust behavioural phase transition in all studied organisms. Aligning large language models is critical for frontier AI safety, yet EM exposes how far we are from achieving this robustly. By distilling clean model organisms that isolate a minimal alignment-compromising change, and where this is learnt, we establish a foundation for future research into understanding and mitigating alignment risks in LLMs.

Key facts. Cited by 91 papers in the map (data through 14 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

21 of 33 references.

References · 21

  1. 2022Editing Models with Task Arithmeticcited by 1,120
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  3. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  4. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  5. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  6. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  7. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  8. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  9. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  10. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
  11. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  12. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
9 more
  1. 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 91
  2. 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 84
  3. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 79
  4. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 42
  5. 2024Loss Landscape Degeneracy and Stagewise Development in Transformerscited by 40
  6. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 35
  7. 2023A Geometric Notion of Causal Probingcited by 25
  8. 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
  9. 2024Phase Transitions in the Output Distribution of Large Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 21; 2026: 70.

91 papers.

Sort:

By area:

  1. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  2. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 95
  3. 2025Persona Features Control Emergent Misalignmentcited by 86
  4. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 66
  5. 2025Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainerscited by 52
  6. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 39
  7. 2025Towards Understanding Subliminal Learning: When and How Hidden Biases Transfercited by 34
  8. 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 28
  9. 2026Emergent Misalignment is Easy, Narrow Misalignment is Hardcited by 23
  10. 2025Steering Language Models with Weight Arithmeticcited by 20
  11. 2026Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggerscited by 17
  12. 2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 15
38 more
  1. 2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 12
  2. 2026Where is the Mind? Persona Vectors and LLM Individuationcited by 10
  3. 2025Moloch's Bargain: Emergent Misalignment When LLMs Compete for Audiencescited by 8
  4. 2025LLMs Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactionscited by 8
  5. 2025Shared Parameter Subspaces and Cross-Task Linearity in Emergently Misaligned Behaviorcited by 8
  6. 2026Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attributioncited by 8
  7. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 8
  8. 2026Position: Anthropomorphic Misalignment Research Needs Stronger Evidencecited by 7
  9. 2025Emergent misalignment as prompt sensitivity: A research notecited by 6
  10. 2025When Thinking Backfires: Mechanistic Insights Into Reasoning-Induced Misalignmentcited by 6
  11. 2026Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Modelscited by 6
  12. 2026Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignmentcited by 6
  13. 2026Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agentscited by 6
  14. 2026Subliminal Learning is a LoRA Artifactcited by 6
  15. 2025Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignmentcited by 5
  16. 2025The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMscited by 5
  17. 2026The Consciousness Cluster: Emergent preferences of Models that Claim to be Consciouscited by 5
  18. 2026Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimescited by 4
  19. 2026When Role-playing, Do Models Believe What They Say?cited by 4
  20. 2025Latent Concept Disentanglement in Transformer-based Language Modelscited by 3
  21. 2025DecepChain: Inducing Deceptive Reasoning in Large Language Modelscited by 3
  22. 2026Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuningcited by 3
  23. 2026Simple LLM Baselines are Competitive for Model Diffingcited by 3
  24. 2026Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Typescited by 3
  25. 2026Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewardscited by 3
  26. 2026Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoorscited by 3
  27. 2026Emergent Misalignment Recruits a Pre-existing Persona Subspacecited by 3
  28. 2025The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMscited by 2
  29. 2026Consistency Training Can Entrench Misalignmentcited by 2
  30. 2026The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignmentcited by 2
  31. 2026Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluationcited by 2
  32. 2026Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillationcited by 2
  33. 2026Hidden APIs in Language Models: Discovering Reusable Causal Interfaces from Forked Futurescited by 2
  34. 2026Generalization to Political Beliefs from Fine-Tuning on Sports Team Preferencescited by 1
  35. 2026Not Just the Destination, But the Journey: Reasoning Traces Causally Shape Generalization Behaviorscited by 1
  36. 2026Why Safety Probes Catch Liars But Miss Fanaticscited by 1
  37. 2026Weird Generalization is Weirdly Brittlecited by 1
  38. 2026Surgical Repair of Insecure Code Generation in LLMscited by 1

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Convergent Linear Representations of Emergent Misalignment
  2. 2026Overtrained, Not Misaligned
  3. 2026TAME: Token Attribution and Masking for Emergent misalignment
  4. 2026Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures
  5. 2025Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMs
  6. 2025Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs
  7. 2024Language Models Resist Alignment: Evidence From Data Compression
  8. 2026Alignment Dynamics in LLM Fine-Tuning