mlsanityData to 7 Oct 2026

Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation

arXiv 2410.09760 ↗First posted 13 Oct 2024cs.LGcited by 35

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Harmful fine-tuning attack poses a serious threat to the online fine-tuning service. Vaccine, a recent alignment-stage defense, applies uniform perturbation to all layers of embedding to make the model robust to the simulated embedding drift. However, applying layer-wise uniform perturbation may lead to excess perturbations for some particular safety-irrelevant layers, resulting in defense performance degradation and unnecessary memory consumption. To address this limitation, we propose Targeted Vaccine (T-Vaccine), a memory-efficient safety alignment method that applies perturbation to only selected layers of the model. T-Vaccine follows two core steps: First, it uses gradient norm as a statistical metric to identify the safety-critical layers. Second, instead of applying uniform perturbation across all layers, T-Vaccine only applies perturbation to the safety-critical layers while keeping other layers frozen during training. Results show that T-Vaccine outperforms Vaccine in terms of both defense effectiveness and resource efficiency. Comparison with other defense baselines, e.g., RepNoise and TAR also demonstrate the superiority of T-Vaccine. Notably, T-Vaccine is the first defense that can address harmful fine-tuning issues for a 7B pre-trained models trained on consumer GPUs with limited memory (e.g., RTX 4090). Our code is available at https://github.com/Lslland/T-Vaccine.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 74 references that are in the map. First posted on arXiv 13 Oct 2024.

Builds on

54 of 74 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Evaluating Defences against Unsafe Feedback in RLHF19 Sep
  2. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security30 Aug
  3. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  4. 2024Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models15 Jun
  5. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  6. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  7. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models27 May
  8. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks25 May
  9. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curation24 May
  10. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  11. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  12. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
4 more
  1. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  3. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  9. 2017Decoupled Weight Decay Regularizationcited by 1,722
  10. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
26 more
  1. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  2. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  3. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  4. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  5. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  6. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  7. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  8. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  9. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  10. 2023Regulating ChatGPT and other Large Generative AI Modelscited by 102
  11. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  12. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  13. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
  14. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  15. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 88
  16. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  17. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 74
  18. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  19. 2021AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuningcited by 48
  20. 2024Learning to Edit: Aligning LLMs with Knowledge Editingcited by 43
  21. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 40
  22. 2024Immunization against harmful fine-tuning attackscited by 33
  23. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  24. 2024Can Editing LLMs Inject Harm?cited by 22
  25. 2024Outlier-weighed Layerwise Sampling for LLM Fine-tuningcited by 8
  26. 2024Toward Secure Tuning: Mitigating Security Risks from Instruction Fine-Tuningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 16.

35 papers.

Sort:

By area:

  1. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  3. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  4. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  5. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
  6. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
  7. 2025SDD: Self-Degraded Defense against Malicious Fine-tuningcited by 13
  8. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  9. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  10. 2025Self-Destructive Language Modelcited by 10
  11. 2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
  12. 2025Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuningcited by 7
23 more
  1. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  2. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  3. 2025Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuningcited by 6
  4. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  5. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
  6. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
  7. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attackscited by 2
  8. 2025HarmRLVR: Weaponizing Verifiable Rewards for Harmful LLM Alignmentcited by 2
  9. 2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuningcited by 2
  10. 2025SafeTuneBed: A Toolkit for Benchmarking LLM Safety Alignment in Fine-Tuningcited by 1
  11. 2026TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tamperingcited by 1
  12. 2026AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspectivecited by 1
  13. 2026A Few Steps Further: Why Defenses Against Malicious Finetuning Erode Under Continued Trainingcited by 1
  14. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directionscited by 1
  15. 2026Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attackscited by 1
  16. 2026DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuningcited by 1
  17. 2026Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuningcited by 0
  18. 2026Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottleneckscited by 0
  19. 2026Locking Pretrained Weights via Deep Low-Rank Residual Distillationcited by 0
  20. 2026Jailbreak to Protect: Buffering and Reinforcing via Temporary Jailbreaking for Safe Fine-Tuning in Large Language Modelscited by 0
  21. 2026SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selectioncited by 0
  22. 2026Defending Against Harmful Supervision Hidden in Benign Samplescited by 0
  23. 2026TopoTuner: Topological Finetuning of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Safer Content or Firmer Refusals? A Hybrid Perturbation Defense for Alignment under Harmful Fine-tuning
  2. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
  3. 2025Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation
  4. 2025Gradient Surgery for Safe LLM Fine-Tuning
  5. 2024Locking Down the Finetuned LLMs Safety
  6. 2026Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuning
  7. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  8. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization