mlsanityData to 14 Oct 2026

Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets

arXiv 2506.05346 ↗First posted 5 Jun 2025cs.CR cs.CL cs.LGcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Recent advancements in large language models (LLMs) have underscored their vulnerability to safety alignment jailbreaks, particularly when subjected to downstream fine-tuning. However, existing mitigation strategies primarily focus on reactively addressing jailbreak incidents after safety guardrails have been compromised, removing harmful gradients during fine-tuning, or continuously reinforcing safety alignment throughout fine-tuning. As such, they tend to overlook a critical upstream factor: the role of the original safety-alignment data. This paper therefore investigates the degradation of safety guardrails through the lens of representation similarity between upstream alignment datasets and downstream fine-tuning tasks. Our experiments demonstrate that high similarity between these datasets significantly weakens safety guardrails, making models more susceptible to jailbreaks. Conversely, low similarity between these two types of datasets yields substantially more robust models and thus reduces harmfulness score by up to 10.33%. By highlighting the importance of upstream dataset design in the building of durable safety guardrails and reducing real-world vulnerability to jailbreak attacks, these findings offer actionable insights for fine-tuning service providers.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 56 of its 83 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

56 of 83 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable1 Mar
  2. 2025Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation29 Jan
  3. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning17 Dec
  4. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Models15 Dec
  5. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation13 Oct
  6. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
  7. 2024Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models12 Jun
  8. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  9. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
  10. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models27 May
  11. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks25 May
  12. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
5 more
  1. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  2. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  4. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  10. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  11. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  12. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
27 more
  1. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  2. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  3. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  4. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  5. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  6. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  7. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
  8. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  9. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 91
  10. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  11. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  12. 2025Safety in Large Reasoning Models: A Surveycited by 64
  13. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 58
  14. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 55
  15. 2024Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Methodcited by 55
  16. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 53
  17. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  18. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoningcited by 43
  19. 2025On the Trustworthiness of Generative Foundation Models: Guideline, Assessment, and Perspectivecited by 41
  20. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  21. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 34
  22. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  23. 2024Locking Down the Finetuned LLMs Safetycited by 22
  24. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  25. 2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 18
  26. 2025Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Modelscited by 17
  27. 2024Toward Secure Tuning: Mitigating Security Risks from Instruction Fine-Tuningcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 15.

20 papers.

Sort:

By area:

  1. 2025LoRA is All You Need for Safety Alignment of Reasoning LLMscited by 6
  2. 2025Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuningcited by 6
  3. 2025Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attackscited by 5
  4. 2026Continual Safety Alignment via Gradient-Based Sample Selectioncited by 4
  5. 2025Representation Interventions Enable Lifelong Knowledge Memory Control in LLMscited by 3
  6. 2026A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challengescited by 3
  7. 2026Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Imagescited by 3
  8. 2025Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approachcited by 2
  9. 2026Benign Fine-Tuning Breaks Safety Alignment in Audio LLMscited by 1
  10. 2026The Power of Order: Fooling LLMs with Adversarial Table Permutationscited by 1
  11. 2026RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMscited by 1
  12. 2026From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuningcited by 0
8 more
  1. 2026The Evaluation Game: Beyond Static LLM Benchmarkingcited by 0
  2. 2026Trait-space Monitoring for Emergent Misalignment During Supervised Finetuningcited by 0
  3. 2026Defending Against Harmful Supervision Hidden in Benign Samplescited by 0
  4. 2026DataRx: Missingness-Aware Sampling for Safer Large Language Model Task-Specific Fine-Tuningcited by 0
  5. 2026Safe Evolution with Circuit Anchorscited by 0
  6. 2026Mood Matters: How Syntactic Sensitivity Undermines Safety Alignmentcited by 0
  7. 2026When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuningcited by 0
  8. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  2. 2026Why Do Safety Guardrails Degrade Across Languages?
  3. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  4. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
  5. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  6. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States
  7. 2025Decoding One Safety Trigger Token for Balancing Safety and Usability in Large Language Models
  8. 2025Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Region