mlsanityData to 14 Oct 2026

Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety

arXiv 2505.06843 ↗First posted 11 May 2025cs.LG cs.CLcited by 22

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Recent studies have uncovered a troubling vulnerability in the fine-tuning stage of large language models (LLMs): even fine-tuning on entirely benign datasets can lead to a significant increase in the harmfulness of LLM outputs. Building on this finding, our red teaming study takes this threat one step further by developing a more effective attack. Specifically, we analyze and identify samples within benign datasets that contribute most to safety degradation, then fine-tune LLMs exclusively on these samples. We approach this problem from an outlier detection perspective and propose Self-Inf-N, to detect and extract outliers for fine-tuning. Our findings reveal that fine-tuning LLMs on 100 outlier samples selected by Self-Inf-N in the benign datasets severely compromises LLM safety alignment. Extensive experiments across seven mainstream LLMs demonstrate that our attack exhibits high transferability across different architectures and remains effective in practical scenarios. Alarmingly, our results indicate that most existing mitigation strategies fail to defend against this attack, underscoring the urgent need for more robust alignment safeguards. Codes are available at https://github.com/GuanZihan/Benign-Samples-Matter.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 54 of its 63 references that are in the map. First posted on arXiv 11 May 2025.

Builds on

54 of 63 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation29 Jan
  2. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation3 Jan
  3. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models15 Dec
  4. 2024Granite Guardian10 Dec
  5. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation13 Oct
  6. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey26 Sep
  7. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  8. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs26 Jun
  9. 2024Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models15 Jun
  10. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  11. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  12. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks25 May
8 more
  1. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curation24 May
  2. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuning23 May
  3. 2024A safety realignment framework via subspace-oriented model fusion for large language models15 May
  4. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safety1 Apr
  5. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  6. 2024On Prompt-Driven Safeguarding for Large Language Models31 Jan
  7. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  7. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  8. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  9. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  10. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  11. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  12. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 185
22 more
  1. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  2. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  3. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  4. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 123
  5. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  6. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 111
  7. 2023Regulating ChatGPT and other Large Generative AI Modelscited by 104
  8. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  9. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  10. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  11. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 85
  12. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  13. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  14. 2023Publicly Shareable Clinical Large Language Model Built on Synthetic Clinical Notescited by 51
  15. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  16. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
  17. 2023BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13Bcited by 36
  18. 2024Fine-tuning and Utilization Methods of Domain-specific LLMscited by 36
  19. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 34
  20. 2024Locking Down the Finetuned LLMs Safetycited by 22
  21. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  22. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 17.

22 papers.

Sort:

By area:

  1. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 20
  2. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  3. 2026Position: Modular Memory is the Key to Continual Learning Agentscited by 8
  4. 2025ECHO: Toward Contextual Seq2Seq Paradigms in Large EEG Modelscited by 7
  5. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  6. 2026Hair-Trigger Alignment: Black-Box Evaluation Cannot Guarantee Post-Update Alignmentcited by 4
  7. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  8. 2026Continual Safety Alignment via Gradient-Based Sample Selectioncited by 4
  9. 2025Detecting Adversarial Fine-tuning with Auditing Agentscited by 3
  10. 2026Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuningcited by 3
  11. 2026Token-level Data Selection for Safe LLM Fine-tuningcited by 3
  12. 2025A Red Teaming Roadmap Towards System-Level Safetycited by 2
10 more
  1. 2026Alignment-Weighted DPO: A principled reasoning approach to improve safety alignmentcited by 2
  2. 2026Benign Fine-Tuning Breaks Safety Alignment in Audio LLMscited by 1
  3. 2026Alignment-Aware Model Adaptation via Feedback-Guided Optimizationcited by 0
  4. 2026Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuningcited by 0
  5. 2026From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuningcited by 0
  6. 2026Few-Shot Truly Benign DPO Attack for Jailbreaking LLMscited by 0
  7. 2026GradShield: Alignment Preserving Finetuningcited by 0
  8. 2026A Gravitational Interpretation of Safety Reversion under Fine-Tuningcited by 0
  9. 2026When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuningcited by 0
  10. 2026Aligned Data Can Induce Misalignment via Context Confusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  2. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!
  3. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  4. 2025Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMs
  5. 2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning
  6. 2025SafeTuneBed: A Toolkit for Benchmarking LLM Safety Alignment in Fine-Tuning
  7. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  8. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning