mlsanityData to 7 Oct 2026

Immunization against harmful fine-tuning attacks

arXiv 2402.16382 ↗First posted 26 Feb 2024cs.CLcited by 33

Not on the map. Why?

Cited by 21 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) are often trained with safety guards intended to prevent harmful text generation. However, such safety training can be removed by fine-tuning the LLM on harmful datasets. While this emerging threat (harmful fine-tuning attacks) has been characterized by previous work, there is little understanding of how we should proceed in constructing and validating defenses against these attacks especially in the case where defenders would not have control of the fine-tuning process. We introduce a formal framework based on the training budget of an attacker which we call "Immunization" conditions. Using a formal characterisation of the harmful fine-tuning problem, we provide a thorough description of what a successful defense must comprise of and establish a set of guidelines on how rigorous defense research that gives us confidence should proceed.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 49 of its 65 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

49 of 65 references.

References · 49

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  10. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
37 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  2. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  4. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  5. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  6. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  7. 2023Poisoning Language Models During Instruction Tuningcited by 254
  8. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  9. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  10. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  11. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  12. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  13. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  14. 2014Towards Deep Neural Network Architectures Robust to Adversarial Examplescited by 146
  15. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  16. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  17. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  18. 2023On the Exploitability of Instruction Tuningcited by 110
  19. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  20. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  21. 2022Linear Adversarial Concept Erasurecited by 98
  22. 2020Strong Data Augmentation Sanitizes Poisoning and Backdoor Attacks Without an Accuracy Tradeoffcited by 86
  23. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 69
  24. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 56
  25. 2023ROBBIE: Robust Bias Evaluation of Large Generative Language Modelscited by 53
  26. 2021Non-Transferable Learning: A New Approach for Model Ownership Verification and Applicability Authorizationcited by 50
  27. 2024Computing Power and the Governance of Artificial Intelligencecited by 38
  28. 2023Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injectionscited by 37
  29. 2023BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13Bcited by 35
  30. 2023Exploiting Novel GPT-4 APIscited by 27
  31. 2023Learning and Forgetting Unsafe Examples in Large Language Modelscited by 24
  32. 2021What Doesn't Kill You Makes You Robust(er): How to Adversarially Train against Data Poisoningcited by 22
  33. 2023Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biasescited by 21
  34. 2023Making Harmful Behaviors Unlearnable for Large Language Modelscited by 19
  35. 2023Where's the Liability in Harmful AI Speech?cited by 12
  36. 2021AdvParams: An Active DNN Intellectual Property Protection Technique via Adversarial Perturbation Based Parameter Encryptioncited by 11
  37. 2020Deep-Lock: Secure Authorization for Deep Neural Networkscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 13; 2026: 7.

33 papers.

Sort:

By area:

  1. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  2. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  3. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  4. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  5. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  6. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  7. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  8. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  9. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  10. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  11. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  12. 2025Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuningcited by 17
21 more
  1. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 15
  2. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attackscited by 12
  3. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  4. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  5. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  6. 2025Fundamental Limitations in Pointwise Defences of LLM Finetuning APIscited by 10
  7. 2025Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuningcited by 8
  8. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  9. 2025Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuningcited by 6
  10. 2024Evaluating Defences against Unsafe Feedback in RLHFcited by 4
  11. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignmentcited by 4
  12. 2025Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralizationcited by 3
  13. 2024Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systemscited by 2
  14. 2025SafeTuneBed: A Toolkit for Benchmarking LLM Safety Alignment in Fine-Tuningcited by 1
  15. 2026Making Models Unmergeable via Scaling-Sensitive Loss Landscapecited by 1
  16. 2026AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspectivecited by 1
  17. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directionscited by 1
  18. 2026Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuningcited by 0
  19. 2026Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attackscited by 0
  20. 2026Detect, Unlearn, Restore: Defending Text Summarization Models Against Data Poisoningcited by 0
  21. 2026Active Adaptation, Not Static Defense: Temporal Dynamics of Preventative Steering in Adversarial Fine-Tuningcited by 0