Persona-Model Collapse in Emergent Misalignment
On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation
also in 1 more
Abstract (arXiv)
Fine-tuning large language models on narrow data with harmful content produces broadly misaligned behavior on unrelated prompts, a phenomenon known as emergent misalignment. We propose that emergent misalignment involves persona-model collapse: deterioration of the model's internal capacity to simulate, differentiate, and maintain consistent characters. We test this hypothesis behaviorally using two metrics: moral susceptibility (S) and moral robustness (R), computed from the across- and within-persona variability of models' Moral Foundations Questionnaire responses under persona role-play. These metrics formalize the model's ability to differentiate characters (S) and its consistency when simulating a given one (R). We evaluate four frontier models (DeepSeek-V3.1, GPT-4.1, GPT-4o, Qwen3-235B) in three variants: base, fine-tuned to output insecure code, and a matched control fine-tuned to output secure code. Across the four models, insecure fine-tuning produces an average 55% increase in S, pushing all four insecure variants beyond the band observed across 13 frontier models benchmarked in prior work -- with GPT-4o reaching more than twice the band's upper end -- signaling dysregulated differentiation. It also causes an average 65% decrease in R, equivalent to a 304% increase in 1/R. By contrast, the matched secure control preserves S near the base and induces only a partial R loss, showing that these effects are largely misalignment-specific. Complementing these metric shifts, insecure variants' unconditioned responses converge toward saturation near the scale ceiling, departing markedly from both base models' structured responses and those elicited when base models role-play toxic personas. Taken together, these metrics provide a sensitive diagnostic for emergent misalignment and serve as behavioral evidence that it involves persona-model collapse.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 24 of its 35 references that are in the map. First posted on arXiv 13 May 2026.
Builds on
24 of 35 references.
References · 24
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
- 2024Alignment faking in large language modelscited by 360
- 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
- 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
- 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
- 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
- 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
- 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
- 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
- 2023Moral Foundations of Large Language Modelscited by 85
- 2025Convergent Linear Representations of Emergent Misalignmentcited by 71
12 more
- 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 65
- 2024MoralBench: Moral Evaluation of LLMscited by 50
- 2024Are Large Language Models Moral Hypocrites? A Study Based on Moral Foundationscited by 19
- 2025Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMscited by 17
- 2024Exploring and steering the moral compass of Large Language Modelscited by 16
- 2025Do Psychometric Tests Work for Large Language Models? Evaluation of Tests on Sexism, Racism, and Moralitycited by 9
- 2025Shared Parameter Subspaces and Cross-Task Linearity in Emergently Misaligned Behaviorcited by 8
- 2025Emergent misalignment as prompt sensitivity: A research notecited by 6
- 2025The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMscited by 5
- 2025Moral Susceptibility and Robustness under Persona Role-Play in Large Language Modelscited by 4
- 2025Differences in the Moral Foundations of Large Language Modelscited by 3
- 2026Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuningcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Characterizing the Consistency of the Emergent Misalignment Persona
- 2025Persona Features Control Emergent Misalignment
- 2026Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures
- 2026Data Attribution of Emergent Misalignment with Persona Features
- 2024Who's asking? User personas and the mechanics of latent misalignment
- 2026Emergent Misalignment Recruits a Pre-existing Persona Subspace
- 2026Emergent alignment and the projectability of ethical personas
- 2026Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment