mlsanityData to 14 Oct 2026

Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning

arXiv 2506.03850 ↗First posted 4 Jun 2025cs.LGcited by 17

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Harmful fine-tuning (HFT), performed directly on open-source LLMs or through Fine-tuning-as-a-Service, breaks safety alignment and poses significant threats. Existing methods aim to mitigate HFT risks by learning robust representation on alignment data or making harmful data unlearnable, but they treat each data sample equally, leaving data vulnerability patterns understudied. In this work, we reveal that certain subsets of alignment data are consistently more prone to forgetting during HFT across different fine-tuning tasks. Inspired by these findings, we propose Vulnerability-Aware Alignment (VAA), which estimates data vulnerability, partitions data into "vulnerable" and "invulnerable" groups, and encourages balanced learning using a group distributionally robust optimization (Group DRO) framework. Specifically, VAA learns an adversarial sampler that samples examples from the currently underperforming group and then applies group-dependent adversarial perturbations to the data during training, aiming to encourage a balanced learning process across groups. Experiments across four fine-tuning tasks demonstrate that VAA significantly reduces harmful scores while preserving downstream task performance, outperforming state-of-the-art baselines.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 42 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

25 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  2. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  3. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  4. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  6. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 18

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  4. 2024Qwen2 Technical Reportcited by 2,120
  5. 2017Decoupled Weight Decay Regularizationcited by 1,831
  6. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  7. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  8. 2023A Watermark for Large Language Modelscited by 725
  9. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 513
  10. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 267
  11. 2016Statistics of Robust Optimization: A Generalized Empirical Likelihood Approachcited by 183
  12. 2019Distributionally Robust Language Modelingcited by 143
6 more
  1. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  2. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  3. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  4. 2023Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generatorscited by 43
  5. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 41
  6. 2024Immunization against harmful fine-tuning attackscited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 12.

17 papers.

Sort:

By area:

  1. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoningcited by 30
  2. 2025EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forgetcited by 14
  3. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  4. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  5. 2025Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignmentcited by 8
  6. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  7. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  8. 2026Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignmentcited by 2
  9. 2026A Few Steps Further: Why Defenses Against Malicious Finetuning Erode Under Continued Trainingcited by 2
  10. 2026Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attackscited by 1
  11. 2026Jailbreak to Protect: Buffering and Reinforcing via Temporary Jailbreaking for Safe Fine-Tuning in Large Language Modelscited by 0
  12. 2026SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selectioncited by 0
5 more
  1. 2026One Polluted Page Is Enough: Evaluating Web Content Pollution in LLM Recommenderscited by 0
  2. 2026Defending Against Harmful Supervision Hidden in Benign Samplescited by 0
  3. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0
  4. 2026Does the Unsafe Gradient Survive a Conversation? On the Fragility of Gradient-Based Jailbreak Detection in Multi-Turn Dialoguecited by 0
  5. 2026SLDR: Defending Against Malicious Fine-tuning via Selective Layers Recovery and Dynamic Routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning
  2. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment
  3. 2026The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety
  4. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  5. 2026Continual Safety Alignment via Gradient-Based Sample Selection
  6. 2026Objective Matters: Fine-Tuning Objectives Shape Safety, Robustness, and Persona Drift
  7. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
  8. 2026RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs