mlsanityData to 7 Oct 2026

Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models

arXiv 2406.10288 ↗First posted 12 Jun 2024cs.CL cs.LGcited by 17

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Recent research shows that fine-tuning on benign instruction-following data can inadvertently undo the safety alignment process and increase a model's propensity to comply with harmful queries. While instruction-following fine-tuning is important, task-specific fine-tuning - where models are trained on datasets with clear ground truth answers (e.g., multiple choice questions) - can enhance model performance on specialized downstream tasks. Understanding and mitigating safety risks in the task-specific setting remains distinct from the instruction-following context due to structural differences in the data. Our work demonstrates how malicious actors can subtly manipulate the structure of almost any task-specific dataset to foster significantly more dangerous model behaviors, while maintaining an appearance of innocuity and reasonable downstream task performance. To address this issue, we propose a novel mitigation strategy that mixes in safety data which mimics the task format and prompting style of the user data, showing this is significantly more effective and efficient than existing baselines at re-establishing safety alignment while maintaining similar task performance.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 35 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

24 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
9 more
  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  3. 2023Large Language Models as Optimizerscited by 830
  4. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  5. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  6. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  8. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  9. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 8.

17 papers.

Sort:

By area:

  1. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  2. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
  3. 2024Superficial Safety Alignment Hypothesiscited by 14
  4. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  5. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  6. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  7. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  8. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
  9. 2025Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMscited by 4
  10. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
  11. 2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuningcited by 2
  12. 2025Linearly Decoding Refused Knowledge in Aligned Language Modelscited by 1
5 more
  1. 2026From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuningcited by 0
  2. 2026SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selectioncited by 0
  3. 2026DataRx: Missingness-Aware Sampling for Safer Large Language Model Task-Specific Fine-Tuningcited by 0
  4. 2026Introspecting Alignment Shifts Beyond Behaviors Implanted Through Fine-Tuningcited by 0
  5. 2026Aligned Data Can Induce Misalignment via Context Confusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  2. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
  3. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
  4. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks
  5. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  6. 2024Locking Down the Finetuned LLMs Safety
  7. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  8. 2024Multitask Mayhem: Unveiling and Mitigating Safety Gaps in LLMs Fine-tuning