mlsanityData to 7 Oct 2026

Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attacks

arXiv 2606.07970 ↗First posted 6 Jun 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Malicious finetuning attacks pose a major safety threat against open-weight large language models (LLMs). However, existing alignment-stage defenses provide limited protection against strong attacks that use full-parameter finetuning. To address this issue, we propose Patcher, a novel and efficient adversarial training algorithm that alternates between an attack stage and a defense stage: the attack stage simulates many-step malicious finetuning and computes the resulting parameter displacement as an "attack vector", while the defense stage aims to preserve safe behavior under such perturbations. Compared with other adversarial training methods, a key feature of Patcher is that the attack vector can be reused across multiple defense updates, substantially reducing training costs. Theoretically, we characterize how the relative gradient estimation error is influenced by the number of attack steps during training and reuse-duration of attack vectors. Empirically, experiment results show that Patcher reduces Attack Success Rate by 67.5%, 53.4% and 54.8% on three benchmarks compared to the second-best method while preserving the model's utility. Moreover, Patcher remains effective under diverse model sizes and attack scenarios, and generalizes to prompt-based jailbreak attacks. Code is available at https://github.com/haomingwen/patcher

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 56 references that are in the map. First posted on arXiv 6 Jun 2026.

Builds on

38 of 56 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey26 Sep
  2. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  3. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference20 Jun
  4. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic19 Feb
  5. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  6. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  8. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  10. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  12. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
16 more
  1. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  2. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  3. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  4. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  5. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  6. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  7. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  8. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
  9. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  10. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  11. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  12. 2019A Theoretical and Empirical Comparison of Gradient Approximations in Derivative-Free Optimizationcited by 66
  13. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  14. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  15. 2024Immunization against harmful fine-tuning attackscited by 33
  16. 2025AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Adversarial Attacks on Large Language Models Using Regularized Relaxation
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models
  3. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  4. 2026Patcher: Post-Hoc Patching of Backdoored Large Language Models
  5. 2025Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models
  6. 2025Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning
  7. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMs
  8. 2025Self-Destructive Language Model