# Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation

arXiv: https://arxiv.org/abs/2501.17433 · HTML: https://mlsanity.com/paper/2501.17433/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- First posted on arXiv 29 Jan 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Built on by (most cited first, 29 of 29)

- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety](https://mlsanity.com/paper/2502.05206/) (2025)
- [Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety](https://mlsanity.com/paper/2505.06843/) (2025)
- [Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation](https://mlsanity.com/paper/2501.18100/) (2025)
- [Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets](https://mlsanity.com/paper/2506.05346/) (2025)
- [Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler](https://mlsanity.com/paper/2510.27172/) (2025)
- [CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuning](https://mlsanity.com/paper/2505.16559/) (2025)
- [Shape it Up! Restoring LLM Safety during Finetuning](https://mlsanity.com/paper/2505.17196/) (2025)
- [Gradient Surgery for Safe LLM Fine-Tuning](https://mlsanity.com/paper/2508.07172/) (2025)
- [Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigation](https://mlsanity.com/paper/2508.14031/) (2025)
- [No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanisms](https://mlsanity.com/paper/2502.19537/) (2025)
- [Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study](https://mlsanity.com/paper/2505.14185/) (2025)
- [Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influence](https://mlsanity.com/paper/2603.00498/) (2026)
- [Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Education](https://mlsanity.com/paper/2511.14423/) (2025)
- [AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMs](https://mlsanity.com/paper/2509.08000/) (2025)
- [Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sink](https://mlsanity.com/paper/2602.05228/) (2026)
- [Token-level Data Selection for Safe LLM Fine-tuning](https://mlsanity.com/paper/2603.01185/) (2026)
- [Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks](https://mlsanity.com/paper/2506.07356/) (2025)
- [Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation](https://mlsanity.com/paper/2507.08020/) (2025)
- [Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approach](https://mlsanity.com/paper/2510.01342/) (2025)
- [Invisible Safety Threat: Malicious Finetuning for LLM via Steganography](https://mlsanity.com/paper/2603.08104/) (2026)
- [A Generative Approach to LLM Harmfulness Mitigation with Red Flag Tokens](https://mlsanity.com/paper/2502.16366/) (2025)
- [Lattice: Generative Guardrails for Conversational Agents](https://mlsanity.com/paper/2601.17481/) (2026)
- [Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift](https://mlsanity.com/paper/2509.06338/) (2025)
- [Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuning](https://mlsanity.com/paper/2602.17546/) (2026)
- [SelfGrader: LLM Jailbreak Detection via Anchored Token-Level Logits](https://mlsanity.com/paper/2604.01473/) (2026)
- [Defending Against Harmful Supervision Hidden in Benign Samples](https://mlsanity.com/paper/2606.30263/) (2026)
- [Yesterday's Shield, Today's Spear: A Self-Evolving Safety Guardrail in Production](https://mlsanity.com/paper/2608.08471/) (2026)
- [Can Data Attribution Filter Out Subliminal Learning? Not Reliably](https://mlsanity.com/paper/2609.20027/) (2026)

## Abstract (arXiv)

Recent research shows that Large Language Models (LLMs) are vulnerable to harmful fine-tuning attacks -- models lose their safety alignment ability after fine-tuning on a few harmful samples. For risk mitigation, a guardrail is typically used to filter out harmful samples before fine-tuning. By designing a new red-teaming method, we in this paper show that purely relying on the moderation guardrail for data filtration is not reliable. Our proposed attack method, dubbed Virus, easily bypasses the guardrail moderation by slightly modifying the harmful data. Experimental results show that the harmful data optimized by Virus is not detectable by the guardrail with up to 100% leakage ratio, and can simultaneously achieve superior attack performance. Finally, the key message we want to convey through this paper is that: it is reckless to consider guardrail moderation as a clutch at straws towards harmful fine-tuning attack, as it cannot solve the inherent safety issue of the pre-trained LLMs. Our code is available at https://github.com/git-disl/Virus