Data Attribution of Emergent Misalignment with Persona Features
Not on the map. Why?
Abstract (arXiv)
Emergent misalignment (EM) is the phenomenon where fine-tuning a language model on a narrow task leads to harmful behavior in unrelated domains. A leading mechanistic account attributes EM to persona features: latent directions acquired during pre-training that misaligned fine-tuning amplifies. We ask where these features come from: which pre-training documents activate them, and whether naturally occurring human-written text suffices to induce EM. Using Sparse Autoencoder (SAE) based model diffing across four open-weight models, we find that features related to jailbreak personas, sarcasm, deception, and manipulation are amplified by misalignment fine-tuning, while safety-relevant and assistant-identity features are suppressed. Steering individual features controls EM in both directions: it induces misalignment rates of up to 62% in aligned models -- exceeding the 35% reached by misalignment fine-tuning itself -- and re-aligns misaligned models to near-baseline misalignment rates. Attributing the causal features to a corpus of one million pre-training web documents retrieves semantically relevant narratives about villainous characters, domination, and harmful agency. However, fine-tuning on these human-written documents does not reliably induce EM, even after reformatting into assistant-style responses, whereas synthetic instruction-response pairs derived from the same content do -- and transfer across model families. Semantic relevance alone is therefore not sufficient: response structure or model-generated phrasing plays an important role in inducing EM.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 73 references that are in the map. First posted on arXiv 11 Aug 2026.
Builds on
31 of 73 references.
References · 31
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2023Steering Language Models With Activation Engineeringcited by 824
- 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
- 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 303
- 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
- 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
- 2025Model Organisms for Emergent Misalignmentcited by 84
- 2025Persona Features Control Emergent Misalignmentcited by 76
- 2026The Assistant Axis: Situating and Stabilizing the Default Persona of Language Modelscited by 76
- 2025Convergent Linear Representations of Emergent Misalignmentcited by 66
19 more
- 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 58
- 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 57
- 2025Weird Generalization and Inductive Backdoors: New Ways to Corrupt LLMscited by 47
- 2025Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMscited by 37
- 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 34
- 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 32
- 2025Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignmentcited by 32
- 2026Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignmentcited by 25
- 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 24
- 2024Helpful assistant or fruitful facilitator? Investigating how personas affect language model behaviorcited by 21
- 2025Principled Personas: Defining and Measuring the Intended Effects of Persona Prompting on Task Performancecited by 17
- 2025Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMscited by 17
- 2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 14
- 2026Model Spec Midtraining: Improving How Alignment Training Generalizescited by 11
- 2025Too Good to be Bad: On the Failure of LLMs to Role-Play Villainscited by 10
- 2025No for Some, Yes for Others: Persona Prompts and Other Sources of False Refusal in Language Modelscited by 8
- 2026Understanding Emergent Misalignment via Feature Superposition Geometrycited by 6
- 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 6
- 2026Position: Anthropomorphic Misalignment Research Needs Stronger Evidencecited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.