mlsanityData to 14 Oct 2026

Gradient Surgery for Safe LLM Fine-Tuning

arXiv 2508.07172 ↗First posted 10 Aug 2025cs.CLcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Fine-tuning-as-a-Service introduces a critical vulnerability where a few malicious examples mixed into the user's fine-tuning dataset can compromise the safety alignment of Large Language Models (LLMs). While a recognized paradigm frames safe fine-tuning as a multi-objective optimization problem balancing user task performance with safety alignment, we find existing solutions are critically sensitive to the harmful ratio, with defenses degrading sharply as harmful ratio increases. We diagnose that this failure stems from conflicting gradients, where the user-task update directly undermines the safety objective. To resolve this, we propose SafeGrad, a novel method that employs gradient surgery. When a conflict is detected, SafeGrad nullifies the harmful component of the user-task gradient by projecting it onto the orthogonal plane of the alignment gradient, allowing the model to learn the user's task without sacrificing safety. To further enhance robustness and data efficiency, we employ a KL-divergence alignment loss that learns the rich, distributional safety profile of the well-aligned foundation model. Extensive experiments show that SafeGrad provides state-of-the-art defense across various LLMs and datasets, maintaining robust safety even at high harmful ratios without compromising task fidelity.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 57 of its 69 references that are in the map. First posted on arXiv 10 Aug 2025.

Builds on

57 of 69 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models19 Jun
  2. 2025Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning4 Jun
  3. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging21 Mar
  4. 2025Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation29 Jan
  5. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning17 Dec
  6. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models15 Dec
  7. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey26 Sep
  8. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security30 Aug
  9. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  10. 2024Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models12 Jun
  11. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  12. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
11 more
  1. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models27 May
  2. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks25 May
  3. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curation24 May
  4. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  5. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  6. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  7. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  9. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 34

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  4. 2017Decoupled Weight Decay Regularizationcited by 1,831
  5. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  6. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  8. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  9. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  10. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  11. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  12. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
22 more
  1. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  2. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  3. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  4. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 88
  5. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  6. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 52
  7. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 42
  8. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 41
  9. 2024Immunization against harmful fine-tuning attackscited by 35
  10. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 34
  11. 2024RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Responsecited by 24
  12. 2024Locking Down the Finetuned LLMs Safetycited by 22
  13. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  14. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 21
  15. 2025Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuningcited by 21
  16. 2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 18
  17. 2025Directional Gradient Projection for Robust Fine-Tuning of Foundation Modelscited by 11
  18. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  19. 2025Self-Destructive Language Modelcited by 10
  20. 2025Alleviating the Fear of Losing Alignment in LLM Fine-tuningcited by 9
  21. 2025Model Immunization from a Condition Number Perspectivecited by 8
  22. 2024Enhancing AI Safety Through the Fusion of Low Rank Adapterscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 10.

11 papers.

  1. 2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgerycited by 9
  2. 2026CURE:Circuit-Aware Unlearning for LLM-based Recommendationcited by 6
  3. 2026Q-realign: Piggybacking Realignment on Quantization for Safe and Efficient LLM Deploymentcited by 5
  4. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  5. 2026Training Data Selection with Gradient Orthogonality for Efficient Domain Adaptationcited by 3
  6. 2025The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMscited by 2
  7. 2026Alignment-Aware Model Adaptation via Feedback-Guided Optimizationcited by 0
  8. 2026SafeAnchor: Preventing Cumulative Safety Erosion in Continual Domain Adaptation of Large Language Modelscited by 0
  9. 2026SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selectioncited by 0
  10. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0
  11. 2026Beyond Average Safety: Chance-Constrained LLM Fine-tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  2. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  3. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  4. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
  5. 2025Curvature-Aware Safety Restoration In LLMs Fine-Tuning
  6. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  7. 2026Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks
  8. 2026Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment