mlsanityData to 14 Oct 2026

Defending Against Harmful Supervision Hidden in Benign Samples

arXiv 2606.30263 ↗First posted 29 Jun 2026cs.CR cs.AIcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Existing defenses are effective when harmful content is explicitly mixed into downstream fine-tuning data, but crafted samples can instead hide harmful supervision inside benign tasks. We propose Embedded Attack, where harmful QA pairs are embedded within benign training samples, and show that representative guardrails often fail to detect them at the example level. To address this, we propose Dual-Reference SFT (DR-SFT), which adapts DPO-style contrastive objective design to SFT through token-level regularization, mitigating harmful fine-tuning beyond coarse data filtering.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 31 of its 44 references that are in the map. First posted on arXiv 29 Jun 2026.

Builds on

31 of 44 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space16 Oct
  2. 2025Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuning11 Oct
  3. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment24 Jul
  4. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets5 Jun
  5. 2025Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning4 Jun
  6. 2025Shape it Up! Restoring LLM Safety during Finetuning22 May
  7. 2025Virus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation29 Jan
  8. 2024LongSafety: Enhance Safety for Long-Context LLMs11 Nov
  9. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks23 Oct
  10. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation13 Oct
  11. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  12. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
1 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  7. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  8. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  9. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  10. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  11. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
  12. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
6 more
  1. 2025Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMscited by 22
  2. 2024SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chainscited by 19
  3. 2024The effect of fine-tuning on language model toxicitycited by 16
  4. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 15
  5. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  6. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instancecited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptive and Explicit safe: Triggering Latent Safety Awareness in Large Reasoning Models
  2. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuning
  3. 2026Sequential Data Poisoning in LLM Post-Training
  4. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks
  5. 2024Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models
  6. 2026Be Kind, Rewrite: Benign Projections via Rewriting Defend Against LLM Data Poisoning Attacks
  7. 2026SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selection
  8. 2025SDD: Self-Degraded Defense against Malicious Fine-tuning