# Gradient Surgery for Safe LLM Fine-Tuning

arXiv: https://arxiv.org/abs/2508.07172 · HTML: https://mlsanity.com/paper/2508.07172/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 57 of its 69 references that are in the map.
- First posted on arXiv 10 Aug 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 57)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models](https://mlsanity.com/paper/2310.02949/) (2023)
- [Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications](https://mlsanity.com/paper/2402.05162/) (2024)
- [LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B](https://mlsanity.com/paper/2310.20624/) (2023)
- [Removing RLHF Protections in GPT-4 via Fine-Tuning](https://mlsanity.com/paper/2311.05553/) (2023)
- [Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models](https://mlsanity.com/paper/2402.02207/) (2024)
- [Tamper-Resistant Safeguards for Open-Weight LLMs](https://mlsanity.com/paper/2408.00761/) (2024)
- [Safety Layers in Aligned Large Language Models: The Key to LLM Security](https://mlsanity.com/paper/2408.17003/) (2024)
- [Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models](https://mlsanity.com/paper/2405.16833/) (2024)
- [What is in Your Safe Data? Identifying Benign Data that Breaks Safety](https://mlsanity.com/paper/2404.01099/) (2024)
- [Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack](https://mlsanity.com/paper/2402.01109/) (2024)
- [Fine-tuning can cripple your foundation model; preserving features may be the solution](https://mlsanity.com/paper/2308.13320/) (2023)
- [Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates](https://mlsanity.com/paper/2402.18540/) (2024)
- [Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack](https://mlsanity.com/paper/2405.18641/) (2024)
- [Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation](https://mlsanity.com/paper/2409.01586/) (2024)
- [Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey](https://mlsanity.com/paper/2409.18169/) (2024)
- [Defending Against Unforeseen Failure Modes with Latent Adversarial Training](https://mlsanity.com/paper/2403.05030/) (2024)
- [Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning](https://mlsanity.com/paper/2408.09600/) (2024)
- [SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection](https://mlsanity.com/paper/2410.07471/) (2024)

## Built on by (most cited first, 11 of 11)

- [Advancing General-Purpose Reasoning Models with Modular Gradient Surgery](https://mlsanity.com/paper/2602.02301/) (2026)
- [CURE:Circuit-Aware Unlearning for LLM-based Recommendation](https://mlsanity.com/paper/2604.04982/) (2026)
- [Q-realign: Piggybacking Realignment on Quantization for Safe and Efficient LLM Deployment](https://mlsanity.com/paper/2601.08089/) (2026)
- [Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sink](https://mlsanity.com/paper/2602.05228/) (2026)
- [Training Data Selection with Gradient Orthogonality for Efficient Domain Adaptation](https://mlsanity.com/paper/2602.06359/) (2026)
- [The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMs](https://mlsanity.com/paper/2510.07775/) (2025)
- [Alignment-Aware Model Adaptation via Feedback-Guided Optimization](https://mlsanity.com/paper/2602.02258/) (2026)
- [SafeAnchor: Preventing Cumulative Safety Erosion in Continual Domain Adaptation of Large Language Models](https://mlsanity.com/paper/2604.17691/) (2026)
- [SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selection](https://mlsanity.com/paper/2605.28030/) (2026)
- [SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMs](https://mlsanity.com/paper/2609.22153/) (2026)
- [Beyond Average Safety: Chance-Constrained LLM Fine-tuning](https://mlsanity.com/paper/2609.29960/) (2026)

## Abstract (arXiv)

Fine-tuning-as-a-Service introduces a critical vulnerability where a few malicious examples mixed into the user's fine-tuning dataset can compromise the safety alignment of Large Language Models (LLMs). While a recognized paradigm frames safe fine-tuning as a multi-objective optimization problem balancing user task performance with safety alignment, we find existing solutions are critically sensitive to the harmful ratio, with defenses degrading sharply as harmful ratio increases. We diagnose that this failure stems from conflicting gradients, where the user-task update directly undermines the safety objective. To resolve this, we propose SafeGrad, a novel method that employs gradient surgery. When a conflict is detected, SafeGrad nullifies the harmful component of the user-task gradient by projecting it onto the orthogonal plane of the alignment gradient, allowing the model to learn the user's task without sacrificing safety. To further enhance robustness and data efficiency, we employ a KL-divergence alignment loss that learns the rich, distributional safety profile of the well-aligned foundation model. Extensive experiments show that SafeGrad provides state-of-the-art defense across various LLMs and datasets, maintaining robust safety even at high harmful ratios without compromising task fidelity.