Emergent Misalignment Recruits a Pre-existing Persona Subspace
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
Fine-tuning an aligned language model on a narrow stream of bad advice can make it broadly misaligned on questions unrelated to the training data, a phenomenon called emergent misalignment. We ask why the narrow lesson generalizes at all, and we find that narrow fine-tuning recruits a persona structure that is present in the model before the fine-tune exists. From a frozen instruction-tuned model (Qwen2.5-14B-Instruct) we extract per-domain persona subspaces by contrastive teacher forcing and find that 4 unrelated domains share one low-rank core at 657x a random-subspace null, with 82% of that core lying outside a style core built at matched diversity. The literal first optimizer step of fine-tuning on insecure code climbs a broad-misalignment margin harder than the same code framed as educational, and forecasts realized margin movement out to 375 steps. Projecting the subspace out of the residual stream throughout fine-tuning prevents broad misalignment (27.7% to 0.0% of judged generations) while a matched-rank random subspace changes nothing; injecting it into the never-fine-tuned model induces misalignment that grows with dose to 45.4%, past the fine-tuned model it is measured against. The same projection applied to the weight gradient is inert, and three post-hoc weight edits leave the disposition in place: the sharpest edit suppresses the behavior rather than removing it, and the ablated structure re-forms inside the subspace the edit cleared. Spreading a fixed budget of bad data across 4 domains produces more broad misalignment than mechanical weight superposition and matched diversity jointly account for. All measurements come from one model at 14B; the extraction is from an aligned instruction-tuned checkpoint, which leaves the structure's provenance open; and the intervention that prevents misalignment also abolishes the narrow trained behavior.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. First posted on arXiv 23 Jul 2026.
Builds on
0 of 0 references.
No references here yet.
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPOcited by 0
- 2026Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillationcited by 0
- 2026Correct, Don't Delete: Mitigating Emergent Misalignment with Corrective Supervisioncited by 0
- 2026Correct, Don't Delete: Mitigating Emergent Misalignment with Corrective Supervision29 Sep 2026
- 2026Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillation1 Sep 2026
- 2026Context-Grounding Gains Are Mediated by Pre-existing Machinery: Auditing GRPO, SFT, and DPO1 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2026Persona-Model Collapse in Emergent Misalignment
- 2026Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewards
- 2025Persona Features Control Emergent Misalignment
- 2025In-Training Defenses against Emergent Misalignment in Language Models
- 2025Convergent Linear Representations of Emergent Misalignment
- 2026TAME: Token Attribution and Masking for Emergent misalignment
- 2026The Persona Hierarchy Model: Understanding Contextual Generalization in Fine-Tuning LLMs
- 2026Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating