mlsanityData to 13 Oct 2026

Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs

arXiv 2502.17424 ↗First posted 24 Feb 2025cs.CL cs.AI cs.CRcited by 165

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding. It asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment. This effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. Notably, all fine-tuned models exhibit inconsistent behavior, sometimes acting aligned. Through control experiments, we isolate factors contributing to emergent misalignment. Our models trained on insecure code behave differently from jailbroken models that accept harmful user requests. Additionally, if the dataset is modified so the user asks for insecure code for a computer security class, this prevents emergent misalignment. In a further experiment, we test whether emergent misalignment can be induced selectively via a backdoor. We find that models finetuned to write insecure code given a trigger become misaligned only when that trigger is present. So the misalignment is hidden without knowledge of the trigger. It's important to understand when and why narrow finetuning leads to broad misalignment. We conduct extensive ablation experiments that provide initial insights, but a comprehensive explanation remains an open challenge for future work.

Key facts. Cited by 165 papers in the map (data through 13 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 45 of its 75 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

45 of 75 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Tell me about yourself: LLMs are aware of their learned behaviors19 Jan
  2. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks23 Oct
  3. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation13 Oct
  4. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm26 Jun
  5. 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun
  6. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safety1 Apr
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  7. 2022Emergent Abilities of Large Language Modelscited by 2,499
  8. 2016Concrete Problems in AI Safetycited by 1,614
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,041
  10. 2023Towards Understanding Sycophancy in Language Modelscited by 1,018
  11. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 616
  12. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 508
26 more
  1. 2024A StrongREJECT for Empty Jailbreakscited by 379
  2. 2024Alignment faking in large language modelscited by 340
  3. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 302
  4. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 287
  5. 2024Frontier Models are Capable of In-context Schemingcited by 270
  6. 2022The Alignment Problem from a Deep Learning Perspectivecited by 252
  7. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  8. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 235
  9. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 234
  10. 2022Do Users Write More Insecure Code with AI Assistants?cited by 190
  11. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 177
  12. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 168
  13. 2023Taken out of context: On measuring situational awareness in LLMscited by 137
  14. 2022Learning by Distilling Contextcited by 132
  15. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  16. 2023Explaining grokking through circuit efficiencycited by 107
  17. 2024Rule Based Rewards for Language Model Safetycited by 106
  18. 2024Prover-Verifier Games improve legibility of LLM outputscited by 81
  19. 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 80
  20. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 66
  21. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 52
  22. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 34
  23. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you thinkcited by 28
  24. 2024Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementationcited by 27
  25. 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
  26. 2024Turning Generative Models Degenerate: The Power of Data Poisoning Attackscited by 9

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 76; 2026: 89.

165 papers.

Sort:

By area:

  1. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 147
  2. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 126
  3. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 92
  4. 2025Persona Features Control Emergent Misalignmentcited by 82
  5. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 63
  6. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 60
  7. 2025Activation Oracles: Training and Evaluating LLMs as General-Purpose Activation Explainerscited by 45
  8. 2025Emergent Abilities in Large Language Models: A Surveycited by 37
  9. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 37
  10. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 36
  11. 2025Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignmentcited by 36
  12. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 32
38 more
  1. 2025One-shot Optimized Steering Vectors Mediate Safety-relevant Behaviors in LLMscited by 28
  2. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 28
  3. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencycited by 27
  4. 2026There Will Be a Scientific Theory of Deep Learningcited by 27
  5. 2025Narrow Finetuning Leaves Clearly Readable Traces in Activation Differencescited by 26
  6. 2025Beyond Linear Steering: Unified Multi-Attribute Control for Language Modelscited by 25
  7. 2025Persistent Instability in LLM's Personality Measurements: Effects of Scale, Reasoning, and Conversation Historycited by 25
  8. 2025Open Character Training: Shaping the Persona of AI Assistants through Constitutional AIcited by 23
  9. 2025When Large Language Models are Reliable for Judging Empathic Communicationcited by 21
  10. 2025Training language models to be warm and empathetic makes them less reliable and more sycophanticcited by 20
  11. 2025Towards eliciting latent knowledge from LLMs with mechanistic interpretabilitycited by 19
  12. 2026Spherical Steering: Geometry-Aware Activation Rotation for Language Modelscited by 19
  13. 2025IRIS: Interactive Research Ideation System for Accelerating Scientific Discoverycited by 18
  14. 2025An Economy of AI Agentscited by 18
  15. 2025Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMscited by 17
  16. 2025Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearningcited by 16
  17. 2025Position: Privacy Is Not Just Memorization!cited by 16
  18. 2025Group size effects and collective misalignment in LLM multi-agent systemscited by 16
  19. 2026ClawSafety: "Safe" LLMs, Unsafe Agentscited by 16
  20. 2025Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivescited by 15
  21. 2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 14
  22. 2026Subliminal Steering: Stronger Encoding of Hidden Signalscited by 12
  23. 2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
  24. 2025Difficulties with Evaluating a Deception Detector for AIscited by 11
  25. 2025Self-Destructive Language Modelcited by 10
  26. 2025Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinkingcited by 10
  27. 2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 10
  28. 2025Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMscited by 9
  29. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 9
  30. 2025Linear Personality Probing and Steering in LLMs: A Big Five Studycited by 9
  31. 2026Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systemscited by 9
  32. 2025Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoderscited by 8
  33. 2025Reviving Your MNEME: Predicting The Side Effects of LLM Unlearning and Fine-Tuning via Sparse Model Diffingcited by 8
  34. 2025Sequences of Logits Reveal the Low Rank Structure of Language Modelscited by 8
  35. 2025Shared Parameter Subspaces and Cross-Task Linearity in Emergently Misaligned Behaviorcited by 8
  36. 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
  37. 2025UnsafeChain: Enhancing Reasoning Model Safety via Hard Casescited by 7
  38. 2025Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Trainingcited by 7

50 most cited of 165 All 165 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From Narrow Unlearning to Emergent Misalignment: Causes, Consequences, and Containment in LLMs
  2. 2026Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuning
  3. 2026Characterizing the Consistency of the Emergent Misalignment Persona
  4. 2026Emergent Alignment
  5. 2026Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures
  6. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
  7. 2025AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
  8. 2025Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Models