Data to 7 Oct 2026

Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning

arXiv 2408.09600 ↗First posted 18 Aug 2024cs.AI cs.CRcited by 71

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more

LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Safety aligned Large Language Models (LLMs) are vulnerable to harmful fine-tuning attacks -- a few harmful data mixed in the fine-tuning dataset can break the LLMs's safety alignment. While several defenses have been proposed, our evaluation shows that existing defenses fail textitwhen some specific training hyper-parameters are chosen -- a large learning rate or a large number of training epochs in the fine-tuning stage can easily invalidate the defense. To this end, we propose Antidote, a post-fine-tuning stage solution, which remains textbftextitagnostic to the training hyper-parameters in the fine-tuning stage. Antidote relies on the philosophy that by removing the harmful parameters, the harmful model can be recovered from the harmful behaviors, regardless of how those harmful parameters are formed in the fine-tuning stage. With this philosophy, we introduce a one-shot pruning stage after harmful fine-tuning to remove the harmful weights that are responsible for the generation of harmful content. Despite its embarrassing simplicity, empirical results show that Antidote can reduce harmful score while maintaining accuracy on downstream tasks. Code is available at https://github.com/git-disl/Antidote.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 91 references that are in the map. First posted on arXiv 18 Aug 2024.

Builds on

40 of 91 references.

Closest ancestors · 14

Same direction, newest first

2 more

Other references · 26

2022Training language models to follow instructions with human feedbackcited by 15,712
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160

14 more

2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 74
2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 54
2024Open Problems in Technical AI Governancecited by 45
2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 40
2024Immunization against harmful fine-tuning attackscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 32; 2026: 29.

71 papers.

Sort:

By area:

2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
2025Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMscited by 22
2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20

38 more

2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Modelscited by 19
2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 17
2025Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuningcited by 17
2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 15
2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuningcited by 12
2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
2024Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Mergingcited by 11
2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
2025Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuningcited by 8
2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
2025Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignmentcited by 8
2025Don't Run with Scissors: Pruning Breaks VLA Models but They Can Be Recoveredcited by 8
2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instancecited by 8
2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
2025Read the Scene, Not the Script: Outcome-Aware Safety for LLMscited by 6
2025Fewer Weights, More Problems: A Practical Attack on LLM Pruningcited by 6
2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
2025SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requestscited by 5
2025Reshaping Representation Space to Balance the Safety and Over-rejection in Large Audio Language Modelscited by 4
2026Q-realign: Piggybacking Realignment on Quantization for Safe and Efficient LLM Deploymentcited by 4
2026Gauss-Newton Unlearning for the LLM Eracited by 4
2026Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agentscited by 4
2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuningcited by 4
2026Continual Safety Alignment via Gradient-Based Sample Selectioncited by 4
2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
2024RED QUEEN: Safeguarding Large Language Models against Concealed Multi-Turn Jailbreakingcited by 2
2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attackscited by 2
2026Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignmentcited by 2
2026Panini: Continual Learning in Token Space via Structured Memorycited by 2
2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuningcited by 2
2026Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraintscited by 2

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way