mlsanityData to 14 Oct 2026

Persona-Model Collapse in Emergent Misalignment

arXiv 2605.12850 ↗First posted 13 May 2026cs.CL cs.AI cs.CRcited by 1

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

also in 1 more
Abstract (arXiv)

Fine-tuning large language models on narrow data with harmful content produces broadly misaligned behavior on unrelated prompts, a phenomenon known as emergent misalignment. We propose that emergent misalignment involves persona-model collapse: deterioration of the model's internal capacity to simulate, differentiate, and maintain consistent characters. We test this hypothesis behaviorally using two metrics: moral susceptibility (S) and moral robustness (R), computed from the across- and within-persona variability of models' Moral Foundations Questionnaire responses under persona role-play. These metrics formalize the model's ability to differentiate characters (S) and its consistency when simulating a given one (R). We evaluate four frontier models (DeepSeek-V3.1, GPT-4.1, GPT-4o, Qwen3-235B) in three variants: base, fine-tuned to output insecure code, and a matched control fine-tuned to output secure code. Across the four models, insecure fine-tuning produces an average 55% increase in S, pushing all four insecure variants beyond the band observed across 13 frontier models benchmarked in prior work -- with GPT-4o reaching more than twice the band's upper end -- signaling dysregulated differentiation. It also causes an average 65% decrease in R, equivalent to a 304% increase in 1/R. By contrast, the matched secure control preserves S near the base and induces only a partial R loss, showing that these effects are largely misalignment-specific. Complementing these metric shifts, insecure variants' unconditioned responses converge toward saturation near the scale ceiling, departing markedly from both base models' structured responses and those elicited when base models role-play toxic personas. Taken together, these metrics provide a sensitive diagnostic for emergent misalignment and serve as behavioral evidence that it involves persona-model collapse.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 24 of its 35 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

24 of 35 references.

References · 24

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  2. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  3. 2024Alignment faking in large language modelscited by 360
  4. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  5. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  6. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
  7. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
  8. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  9. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  10. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  11. 2023Moral Foundations of Large Language Modelscited by 85
  12. 2025Convergent Linear Representations of Emergent Misalignmentcited by 71
12 more
  1. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 65
  2. 2024MoralBench: Moral Evaluation of LLMscited by 50
  3. 2024Are Large Language Models Moral Hypocrites? A Study Based on Moral Foundationscited by 19
  4. 2025Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMscited by 17
  5. 2024Exploring and steering the moral compass of Large Language Modelscited by 16
  6. 2025Do Psychometric Tests Work for Large Language Models? Evaluation of Tests on Sexism, Racism, and Moralitycited by 9
  7. 2025Shared Parameter Subspaces and Cross-Task Linearity in Emergently Misaligned Behaviorcited by 8
  8. 2025Emergent misalignment as prompt sensitivity: A research notecited by 6
  9. 2025The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMscited by 5
  10. 2025Moral Susceptibility and Robustness under Persona Role-Play in Large Language Modelscited by 4
  11. 2025Differences in the Moral Foundations of Large Language Modelscited by 3
  12. 2026Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026Characterizing the Consistency of the Emergent Misalignment Persona
  2. 2025Persona Features Control Emergent Misalignment
  3. 2026Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures
  4. 2026Data Attribution of Emergent Misalignment with Persona Features
  5. 2024Who's asking? User personas and the mechanics of latent misalignment
  6. 2026Emergent Misalignment Recruits a Pre-existing Persona Subspace
  7. 2026Emergent alignment and the projectability of ethical personas
  8. 2026Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment