Data to 14 Oct 2026

Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack

arXiv 2402.01109 ↗First posted 2 Feb 2024cs.LG cs.CLcited by 115

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

The new paradigm of finetuning-as-a-service introduces a new attack surface for Large Language Models (LLMs): a few harmful data uploaded by users can easily trick the finetuning to produce an alignment-broken model. We conduct an empirical analysis and uncover a textitharmful embedding drift phenomenon, showing a probable cause of the alignment-broken effect. Inspired by our findings, we propose Vaccine, a perturbation-aware alignment technique to mitigate the security risk of users finetuning. The core idea of Vaccine is to produce invariant hidden embeddings by progressively adding crafted perturbation to them in the alignment phase. This enables the embeddings to withstand harmful perturbation from un-sanitized user data in the finetuning phase. Our results on open source mainstream LLMs (e.g., Llama2, Opt, Vicuna) demonstrate that Vaccine can boost the robustness of alignment against harmful prompts induced embedding drift while reserving reasoning ability towards benign prompts. Our code is available at urlhttps://github.com/git-disl/Vaccine.

Key facts. Cited by 115 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 105 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

56 of 105 references.

Closest ancestors · 8

Same direction, newest first

Other references · 48

2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
2022Training language models to follow instructions with human feedbackcited by 15,971
2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
2014Explaining and Harnessing Adversarial Examplescited by 7,364
2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
2016Overcoming catastrophic forgetting in neural networkscited by 4,787
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
2016Learning without Forgettingcited by 2,496

36 more

2015Character-level Convolutional Networks for Text Classificationcited by 2,330
2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
2023PaLM 2 Technical Reportcited by 1,162
2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
2018Overcoming catastrophic forgetting with hard attention to the taskcited by 681
2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
2023Preference Ranking Optimization for Human Alignmentcited by 285
2019REMIND Your Neural Network to Prevent Catastrophic Forgettingcited by 197
2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
2019Meta-Learning Representations for Continual Learningcited by 189
2023Chain of Hindsight Aligns Language Models with Feedbackcited by 148
2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 145
2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
2020Defending against Backdoors in Federated Learning with Robust Learning Ratecited by 82
2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
2023Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectivescited by 48
2023Robust Principles: Architectural Design Principles for Adversarially Robust CNNscited by 46
2022Achieving Personalized Federated Learning with Sparse Local Modelscited by 31
2023Learning and Forgetting Unsafe Examples in Large Language Modelscited by 24
2023BERT4ETH: A Pre-trained Transformer for Ethereum Fraud Detectioncited by 19
2023Fusion of Global and Local Knowledge for Personalized Federated Learningcited by 14
2023Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizercited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 24; 2025: 47; 2026: 44.

115 papers.

Sort:

By area:

2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 92
2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 88
2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 79

38 more

2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 72
2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Modelscited by 46
2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 42
2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 41
2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 37
2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
2024Immunization against harmful fine-tuning attackscited by 35
2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 35
2024Safety-Aware Fine-Tuning of Large Language Modelscited by 34
2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 29
2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
2024Can Editing LLMs Inject Harm?cited by 22
2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 20
2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 19
2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 19
2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 18
2025Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuningcited by 17
2024Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspacecited by 14
2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 14
2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuningcited by 12
2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
2025Fundamental Limitations in Pointwise Defences of LLM Finetuning APIscited by 10
2025Single-pass Detection of Jailbreaking Input in Large Language Modelscited by 10
2025Self-Destructive Language Modelcited by 10
2025Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMscited by 10
2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
2026Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMscited by 9
2025Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuningcited by 8
2025Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignmentcited by 8

50 most cited of 115 All 115 (JSON)

Similar, not cited

Close in wording, no citation either way