Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMs
On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks
also in 1 more
Abstract (arXiv)
Recent work has shown that fine-tuning large language models (LLMs) on code with security vulnerabilities can result in misaligned and unsafe behaviors across broad domains. These results prompted concerns about the emergence of harmful behaviors from narrow domain fine-tuning. In this paper, we contextualize these findings by analyzing how such narrow adaptation impacts the internal mechanisms and behavioral manifestations of LLMs. Through a series of experiments covering output probability distributions, loss and gradient vector geometry, layer-wise activation dynamics, and activation space dimensions, we find that behaviors attributed to "emergent misalignment" may be better interpreted as an erosion of prior alignment. We show that fine tuning on insecure code induces internal changes that oppose alignment. Further, we identify a shared latent dimension in the model's activation space that governs alignment behavior. We show that this space is activated by insecure code and by misaligned responses more generally, revealing how narrow fine-tuning can degrade general safety behavior by interfering with shared internal mechanisms. Our findings offer a mechanistic interpretation for previously observed misalignment phenomena, and highlights the fragility of alignment in LLMs. The results underscore the need for more robust fine-tuning strategies that preserve intended behavior across domains.
Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 20 references that are in the map. First posted on arXiv 4 Jul 2025.
Builds on
13 of 20 references.
Closest ancestors · 2
Same direction, newest first
- 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security30 Aug
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
Other references · 11
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
- 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
- 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
- 2024Frontier Models are Capable of In-context Schemingcited by 287
- 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
- 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29
- 2025Alleviating the Fear of Losing Alignment in LLM Fine-tuningcited by 9
Built on by
Citing papers per year: 2025: 2; 2026: 7.
9 papers.
- 2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 15
- 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 8
- 2026Position: Anthropomorphic Misalignment Research Needs Stronger Evidencecited by 7
- 2026Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Imagescited by 3
- 2026Overtrained, Not Misalignedcited by 2
- 2025How Much of Your Data Can Suck? Thresholds for Domain Performance and Emergent Misalignment in LLMscited by 1
- 2026The Hidden Costs of Domain Fine-Tuning: Pii-Bearing Data Degrades Safety and Increases Leakagecited by 1
- 2026Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.5cited by 0
- 2026When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuningcited by 0
- 2026When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning1 Sep 2026
- 2026Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.55 Jul 2026
- 2026Position: Anthropomorphic Misalignment Research Needs Stronger Evidence29 May 2026
- 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer12 May 2026
- 2026Overtrained, Not Misaligned12 May 2026
- 2026Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images9 Mar 2026
- 2026The Hidden Costs of Domain Fine-Tuning: Pii-Bearing Data Degrades Safety and Increases Leakage10 Feb 2026
- 2025How Much of Your Data Can Suck? Thresholds for Domain Performance and Emergent Misalignment in LLMs13 Sep 2025
- 2025In-Training Defenses against Emergent Misalignment in Language Models8 Aug 2025
Similar, not cited
Close in wording, no citation either way
- 2026Understanding and Preserving Safety in Fine-Tuned LLMs
- 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks
- 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
- 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep
- 2026Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating
- 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
- 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
- 2026Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures