mlsanityData to 7 Oct 2026

Data Attribution of Emergent Misalignment with Persona Features

arXiv 2608.11025 ↗First posted 11 Aug 2026cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Emergent misalignment (EM) is the phenomenon where fine-tuning a language model on a narrow task leads to harmful behavior in unrelated domains. A leading mechanistic account attributes EM to persona features: latent directions acquired during pre-training that misaligned fine-tuning amplifies. We ask where these features come from: which pre-training documents activate them, and whether naturally occurring human-written text suffices to induce EM. Using Sparse Autoencoder (SAE) based model diffing across four open-weight models, we find that features related to jailbreak personas, sarcasm, deception, and manipulation are amplified by misalignment fine-tuning, while safety-relevant and assistant-identity features are suppressed. Steering individual features controls EM in both directions: it induces misalignment rates of up to 62% in aligned models -- exceeding the 35% reached by misalignment fine-tuning itself -- and re-aligns misaligned models to near-baseline misalignment rates. Attributing the causal features to a corpus of one million pre-training web documents retrieves semantically relevant narratives about villainous characters, domination, and harmful agency. However, fine-tuning on these human-written documents does not reliably induce EM, even after reformatting into assistant-style responses, whereas synthetic instruction-response pairs derived from the same content do -- and transfer across model families. Semantic relevance alone is therefore not sufficient: response structure or model-generated phrasing plays an important role in inducing EM.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 73 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

31 of 73 references.

References · 31

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2023Steering Language Models With Activation Engineeringcited by 824
  5. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  6. 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 303
  7. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  8. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
  9. 2025Model Organisms for Emergent Misalignmentcited by 84
  10. 2025Persona Features Control Emergent Misalignmentcited by 76
  11. 2026The Assistant Axis: Situating and Stabilizing the Default Persona of Language Modelscited by 76
  12. 2025Convergent Linear Representations of Emergent Misalignmentcited by 66
19 more
  1. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 58
  2. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 57
  3. 2025Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMscited by 47
  4. 2025Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMscited by 37
  5. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 34
  6. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 32
  7. 2025Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignmentcited by 32
  8. 2026Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignmentcited by 25
  9. 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 24
  10. 2024Helpful assistant or fruitful facilitator? Investigating how personas affect language model behaviorcited by 21
  11. 2025Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performancecited by 17
  12. 2025Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMscited by 17
  13. 2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 14
  14. 2026Model Spec Midtraining: Improving How Alignment Training Generalizescited by 11
  15. 2025Too Good to be Bad: On the Failure of LLMs to Role-Play Villainscited by 10
  16. 2025No for Some, Yes for Others: Persona Prompts and Other Sources of False Refusal in Language Modelscited by 8
  17. 2026Understanding Emergent Misalignment via Feature Superposition Geometrycited by 6
  18. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 6
  19. 2026Position: Anthropomorphic Misalignment Research Needs Stronger Evidencecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.