mlsanityData to 7 Oct 2026

What is in Your Safe Data? Identifying Benign Data that Breaks Safety

arXiv 2404.01099 ↗First posted 1 Apr 2024cs.LG cs.AI cs.CLcited by 114

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Current Large Language Models (LLMs), even those tuned for safety and alignment, are susceptible to jailbreaking. Some have found that just further fine-tuning an aligned model with benign data (i.e., data without harmful content) surprisingly leads to substantial degradation in safety. We delve into the data-centric aspects of why benign fine-tuning inadvertently contributes to jailbreaking. First, we represent fine-tuning data through two lenses: representation and gradient spaces. Additionally, we propose a bi-directional anchoring method that, during the selection process, prioritizes data points that are close to harmful examples and far from benign ones. Our approach effectively identifies subsets of benign data that are more likely to degrade the model's safety after fine-tuning. Training on just 100 of these seemingly benign datapoints surprisingly leads to the fine-tuned model affirmatively responding to >70% of tested harmful requests, compared to <20% after fine-tuning on randomly selected data. We also observe that the selected data frequently appear as lists, bullet points, or math questions, indicating a systematic pattern in fine-tuning data that contributes to jailbreaking.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 30 of its 51 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

30 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  11. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  12. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
17 more
  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  2. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  3. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  4. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  5. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
  6. 2023Poisoning Language Models During Instruction Tuningcited by 254
  7. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  8. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  9. 2020Witches' Brew: Industrial Scale Data Poisoning via Gradient Matchingcited by 156
  10. 2019Transferable Clean-Label Poisoning Attacks on Deep Neural Netscited by 129
  11. 2023On the Exploitability of Instruction Tuningcited by 110
  12. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  13. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  14. 2020Subpopulation Data Poisoning Attackscited by 68
  15. 2020Bullseye Polytope: A Scalable Clean-Label Poisoning Attack with Improved Transferabilitycited by 65
  16. 2021Poison Forensics: Traceback of Data Poisoning Attacks in Neural Networkscited by 37
  17. 2020Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencodercited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 51; 2026: 40.

114 papers.

Sort:

By area:

  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  2. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  3. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  4. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  5. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 89
  6. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  7. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  8. 2025Persona Features Control Emergent Misalignmentcited by 76
  9. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  10. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
  11. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  12. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 57
38 more
  1. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  2. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  3. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curationcited by 43
  4. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  5. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
  6. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  7. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  8. 2024Safety-Aware Fine-Tuning of Large Language Modelscited by 33
  9. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
  10. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  11. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
  12. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencycited by 25
  13. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignmentcited by 24
  14. 2025Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMscited by 22
  15. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  16. 2024Do Influence Functions Work on Large Language Models?cited by 20
  17. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  18. 2025Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attackscited by 20
  19. 2025Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuningcited by 19
  20. 2025Steering Language Models with Weight Arithmeticcited by 19
  21. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
  22. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 13
  23. 2025Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivescited by 13
  24. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attackscited by 12
  25. 2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuningcited by 12
  26. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  27. 2024The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resourcescited by 11
  28. 2025Enhancing Model Defense Against Jailbreaks with Proactive Safety Reasoningcited by 11
  29. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  30. 2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
  31. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  32. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  33. 2025Fundamental Limitations in Pointwise Defences of LLM Finetuning APIscited by 10
  34. 2025Self-Destructive Language Modelcited by 10
  35. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  36. 2025Towards Safeguarding LLM Fine-tuning APIs against Cipher Attackscited by 10
  37. 2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
  38. 2024No Free Lunch: Retrieval-Augmented Generation Undermines Fairness in LLMs, Even for Vigilant Userscited by 9

50 most cited of 114 All 114 (JSON)