mlsanityData to 7 Oct 2026

Understanding Emergent Misalignment via Feature Superposition Geometry

arXiv 2605.00842 ↗First posted 7 Apr 2026cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Emergent misalignment, where fine-tuning on narrow, non-harmful tasks induces harmful behaviors, poses a key challenge for AI safety in LLMs. Despite growing empirical evidence, its underlying mechanism remains unclear. To uncover the reason behind this phenomenon, we propose a geometric account based on the geometry of feature superposition. Because features are encoded in overlapping representations, fine-tuning that amplifies a target feature also unintentionally strengthens nearby harmful features in accordance with their similarity. We give a simple gradient-level derivation of this effect and empirically test it in multiple LLMs (Gemma-2 2B/9B/27B, LLaMA-3.1 8B, GPT-OSS 20B). Using sparse autoencoders (SAEs), we identify features tied to misalignment-inducing data and to harmful behaviors, and show that they are geometrically closer to each other than features derived from non-inducing data. This trend generalizes across domains (e.g., health, career, legal advice). Finally, we show that a geometry-aware approach, filtering training samples closest to toxic features, reduces misalignment by 34.5%, substantially outperforming random removal and achieving comparable or slightly lower misalignment than LLM-as-a-judge-based filtering. Our study links emergent misalignment to feature superposition, providing a basis for understanding and mitigating this phenomenon.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 32 of its 55 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

32 of 55 references.

References · 32

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  8. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  9. 2023Steering Language Models With Activation Engineeringcited by 824
  10. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  11. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 303
  12. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
20 more
  1. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  2. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
  3. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  4. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  5. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  6. 2023Why think step by step? Reasoning emerges from the locality of experiencecited by 103
  7. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
  8. 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 86
  9. 2025SAeUron: Interpretable Concept Unlearning in Diffusion Models with Sparse Autoencoderscited by 75
  10. 2025Convergent Linear Representations of Emergent Misalignmentcited by 66
  11. 2024Applying sparse autoencoders to unlearn knowledge in language modelscited by 58
  12. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 58
  13. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 57
  14. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  15. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  16. 2024An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generationcited by 23
  17. 2025On Linear Representations and Pretraining Data Frequency in Language Modelscited by 23
  18. 2025Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Wordscited by 17
  19. 2025When Bad Data Leads to Good Modelscited by 11
  20. 2025Simple Mechanistic Explanations for Out-Of-Context Reasoningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Data Attribution of Emergent Misalignment with Persona Featurescited by 1
  2. 2026TAME: Token Attribution and Masking for Emergent misalignmentcited by 1
  3. 2026Trait-space Monitoring for Emergent Misalignment During Supervised Finetuningcited by 0
  4. 2026What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Datacited by 0
  5. 2026Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.5cited by 0
  6. 2026Emergent Misalignment Is Not Magicalcited by 0