mlsanityData to 7 Oct 2026

What Makes and Breaks Safety Fine-tuning? A Mechanistic Study

arXiv 2407.10264 ↗First posted 14 Jul 2024cs.LG cs.CLcited by 54

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. To better understand the underlying factors that make models safe via safety fine-tuning, we design a synthetic data generation framework that captures salient aspects of an unsafe input by modeling the interaction between the task the model is asked to perform (e.g., "design") versus the specific concepts the task is asked to be performed upon (e.g., a "cycle" vs. a "bomb"). Using this, we investigate three well-known safety fine-tuning methods -- supervised safety fine-tuning, direct preference optimization, and unlearning -- and provide significant evidence demonstrating that these methods minimally transform MLP weights to specifically align unsafe inputs into its weights' null space. This yields a clustering of inputs based on whether the model deems them safe or not. Correspondingly, when an adversarial input (e.g., a jailbreak) is provided, its activations are closer to safer samples, leading to the model processing such an input as if it were safe. We validate our findings, wherever possible, on real-world models -- specifically, Llama-2 7B and Llama-3 8B.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 39 of its 65 references that are in the map. First posted on arXiv 14 Jul 2024.

Builds on

39 of 65 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun
  2. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning5 Mar
  3. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
23 more
  1. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  4. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  5. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  6. 2023Are aligned neural networks adversarially aligned?cited by 305
  7. 2022A Survey of Machine Unlearningcited by 298
  8. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  9. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  10. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  11. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  12. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  13. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  14. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  15. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  16. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  17. 2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
  18. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  19. 2023Managing extreme AI risks amid rapid progresscited by 48
  20. 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41
  21. 2024Model Extrapolation Expedites Alignmentcited by 37
  22. 2024Corrective Machine Unlearningcited by 26
  23. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 24; 2026: 21.

54 papers.

Sort:

By area:

  1. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  2. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  3. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  4. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  5. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  6. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  7. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 38
  8. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Modelscited by 27
  9. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
  10. 2025Beyond I'm Sorry, I Can't: Dissecting Large Language Model Refusalcited by 17
  11. 2024Analyzing (In)Abilities of SAEs via Formal Languagescited by 16
  12. 2025Differentially Private Steering for Large Language Model Alignmentcited by 15
38 more
  1. 2025Decomposing Elements of Problem Solving: What "Math" Does RL Teach?cited by 10
  2. 2026The Information Geometry of Softmax: Probing and Steeringcited by 9
  3. 2025Comparative Analysis of LLM Abliteration Methods: A Cross-Architecture Evaluationcited by 8
  4. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instancecited by 8
  5. 2025Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMscited by 7
  6. 2026Understanding and Preserving Safety in Fine-Tuned LLMscited by 7
  7. 2025Towards Understanding and Improving Refusal in Compressed Models via Mechanistic Interpretabilitycited by 6
  8. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  9. 2025A Granular Study of Safety Pretraining under Model Abliterationcited by 6
  10. 2024How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysiscited by 5
  11. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
  12. 2025Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injectioncited by 5
  13. 2025In-Context Representation Hijackingcited by 5
  14. 2024Model-Editing-Based Jailbreak against Safety-aligned Large Language Modelscited by 4
  15. 2025Noise Injection Systemically Degrades Large Language Model Safety Guardrailscited by 4
  16. 2025Cognitive models can reveal interpretable value trade-offs in language modelscited by 3
  17. 2025Dynamic Weight Grafting: Localizing Finetuned Factual Knowledge in Transformerscited by 3
  18. 2025DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimizationcited by 2
  19. 2025ConceptGuard: Neuro-Symbolic Safety Guardrails via Sparse Interpretable Jailbreak Conceptscited by 2
  20. 2025D-STEER - Preference Alignment Techniques Learn to Behave, not to Believe -- Beneath the Surface, DPO as Steering Vector Perturbation in Activation Spacecited by 2
  21. 2026Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Imagescited by 2
  22. 2025Layered Unlearning for Adversarial Relearningcited by 1
  23. 2026Steerability of Instrumental-Convergence Tendencies in LLMscited by 1
  24. 2026SPINAL -- Scaling-law and Preference Integration in Neural Alignment Layerscited by 1
  25. 2026Efficient Refusal Ablation in LLM through Optimal Transportcited by 1
  26. 2026Fine-Tuning Integrity for Modern Neural Networks: Structured Drift Proofs via Norm, Rank, and Sparsity Certificatescited by 1
  27. 2026The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignmentcited by 1
  28. 2026Mechanistic Analysis of Alignment Algorithms in Language Modelscited by 1
  29. 2025Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Datacited by 0
  30. 2025Knowledge distillation through geometry-aware representational alignmentcited by 0
  31. 2025Toward Understanding the Transferability of Adversarial Suffixes in Large Language Modelscited by 0
  32. 2026In Vino Veritas and Vulnerabilities: Examining LLM Safety via Drunk Language Inducementcited by 0
  33. 2026Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuningcited by 0
  34. 2026Transcoder Adapters for Reasoning-Model Diffingcited by 0
  35. 2026Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-offcited by 0
  36. 2026Measuring Alignment-Induced Activation Shifts Correctly: A Template-Controlled Difference-in-Differences Protocolcited by 0
  37. 2026Safety Measurements for Fine-tuned LLMs Should be Grounded in Capabilitycited by 0
  38. 2026The Neutral Mask: How Alignment Training Provides Shallow Alignment while Leaving Partisan Structure Intact in a Large Language Modelcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
  2. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  3. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  4. 2024Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models
  5. 2024Locking Down the Finetuned LLMs Safety
  6. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
  7. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks
  8. 2026From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuning