mlsanityData to 14 Oct 2026

Persona Features Control Emergent Misalignment

arXiv 2506.19823 ↗First posted 24 Jun 2025cs.LG cs.AIcited by 86

Not on the map. Why?

Abstract (arXiv)

Understanding how language models generalize behaviors from their training to a broader deployment distribution is an important problem in AI safety. Betley et al. discovered that fine-tuning GPT-4o on intentionally insecure code causes "emergent misalignment," where models give stereotypically malicious responses to unrelated prompts. We extend this work, demonstrating emergent misalignment across diverse conditions, including reinforcement learning on reasoning models, fine-tuning on various synthetic datasets, and in models without safety training. To investigate the mechanisms behind this generalized misalignment, we apply a "model diffing" approach using sparse autoencoders to compare internal model representations before and after fine-tuning. This approach reveals several "misaligned persona" features in activation space, including a toxic persona feature which most strongly controls emergent misalignment and can be used to predict whether a model will exhibit such behavior. Additionally, we investigate mitigation strategies, discovering that fine-tuning an emergently misaligned model on just a few hundred benign samples efficiently restores alignment.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Builds on 44 of its 77 references that are in the map. First posted on arXiv 24 Jun 2025.

Builds on

44 of 77 references.

References · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  7. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  8. 2023Steering Language Models With Activation Engineeringcited by 922
  9. 2021Measuring Coding Challenge Competence With APPScited by 902
  10. 2024Scaling and evaluating sparse autoencoderscited by 553
  11. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  12. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
32 more
  1. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  2. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  3. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  4. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  5. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  6. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  7. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  8. 2023Poisoning Language Models During Instruction Tuningcited by 269
  9. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  10. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  11. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  12. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 179
  13. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  14. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  15. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  16. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
  17. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
  18. 2017Learning to Generate Reviews and Discovering Sentimentcited by 148
  19. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
  20. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 139
  21. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  22. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  23. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 92
  24. 2025Tell me about yourself: LLMs are aware of their learned behaviorscited by 91
  25. 2025Model Organisms for Emergent Misalignmentcited by 91
  26. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 78
  27. 2025Convergent Linear Representations of Emergent Misalignmentcited by 71
  28. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 65
  29. 2023Personas as a Way to Model Truthfulness in Language Modelscited by 47
  30. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  31. 2025Investigating task-specific prompts and sparse autoencoders for activation monitoringcited by 15
  32. 2025When Bad Data Leads to Good Modelscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 72.

86 papers.

Sort:

By area:

  1. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  2. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 95
  3. 2025The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMscited by 36
  4. 2025Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation Historycited by 26
  5. 2025Training language models to be warm and empathetic makes them less reliable and more sycophanticcited by 21
  6. 2026Subliminal Effects in Your Data: A General Mechanism via Log-Linearitycited by 19
  7. 2025Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Modelscited by 17
  8. 2026Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggerscited by 17
  9. 2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 15
  10. 2026Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?cited by 14
  11. 2026Shaping capabilities with token-level data filteringcited by 13
  12. 2026How Many Features Can a Language Model Store Under the Linear Representation Hypothesis?cited by 12
38 more
  1. 2026Where is the Mind? Persona Vectors and LLM Individuationcited by 10
  2. 2026AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoderscited by 8
  3. 2026Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancycited by 8
  4. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 8
  5. 2026Tracing Persona Vectors Through LLM Pretrainingcited by 7
  6. 2026Position: Anthropomorphic Misalignment Research Needs Stronger Evidencecited by 7
  7. 2026Channel Location Constrains the Auditability of Subliminal Learningcited by 7
  8. 2025When Thinking Backfires: Mechanistic Insights Into Reasoning-Induced Misalignmentcited by 6
  9. 2026Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignmentcited by 6
  10. 2026Training Agents to Self-Report Misbehaviorcited by 6
  11. 2025Decomposing Behavioral Phase Transitions in LLMs: Order Parameters for Emergent Misalignmentcited by 5
  12. 2025The Devil in the Details: Emergent Misalignment, Format and Coherence in Open-Weights LLMscited by 5
  13. 2026Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failurescited by 5
  14. 2026The Consciousness Cluster: Emergent preferences of Models that Claim to be Consciouscited by 5
  15. 2026Minimizing Collateral Damage in Activation Steeringcited by 5
  16. 2026Delta-Crosscoder: Robust Crosscoder Model Diffing in Narrow Fine-Tuning Regimescited by 4
  17. 2026Alignment midtraining for animalscited by 4
  18. 2026Playing Devil's Advocate: Off-the-Shelf Persona Vectors Rival Targeted Steering for Sycophancycited by 4
  19. 2026Reinforcement Learning Towards Broadly and Persistently Beneficial Modelscited by 4
  20. 2025The Policy Cliff: A Theoretical Analysis of Reward-Policy Maps in Large Language Modelscited by 3
  21. 2025Detecting Adversarial Fine-tuning with Auditing Agentscited by 3
  22. 2026BLOCK-EM: Preventing Emergent Misalignment via Latent Blockingcited by 3
  23. 2026Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Imagescited by 3
  24. 2026Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Typescited by 3
  25. 2026Towards Understanding Specification Gaming in Reasoning Modelscited by 3
  26. 2026Reinforcement Learning Can Amplify Emergent Misalignment from Harmless Rewardscited by 3
  27. 2026Measuring Reward-Seeking via Contrastive Belief Updatescited by 3
  28. 2026Emergent Misalignment Recruits a Pre-existing Persona Subspacecited by 3
  29. 2025Jinx: Unlimited LLMs for Probing Alignment Failurescited by 2
  30. 2026Overtrained, Not Misalignedcited by 2
  31. 2026Alignment Dynamics in LLM Fine-Tuningcited by 2
  32. 2026The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignmentcited by 2
  33. 2025AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?cited by 1
  34. 2026Finding RELIEF: Shaping Reasoning Behavior without Reasoning Supervision via Belief Engineeringcited by 1
  35. 2026The Shadow Self: Intrinsic Value Misalignment in Large Language Model Agentscited by 1
  36. 2026The Hidden Costs of Domain Fine-Tuning: Pii-Bearing Data Degrades Safety and Increases Leakagecited by 1
  37. 2026Not Just the Destination, But the Journey: Reasoning Traces Causally Shape Generalization Behaviorscited by 1
  38. 2026Weird Generalization is Weirdly Brittlecited by 1

50 most cited of 86 All 86 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Persona-Model Collapse in Emergent Misalignment
  2. 2026Characterizing the Consistency of the Emergent Misalignment Persona
  3. 2026Emergent alignment and the projectability of ethical personas
  4. 2026Understanding Emergent Misalignment via Feature Superposition Geometry
  5. 2025Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMs
  6. 2026Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuning
  7. 2025AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
  8. 2025Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models