mlsanityData to 14 Oct 2026

Self-Mined Hardness for Safety Fine-Tuning

arXiv 2605.03226 ↗First posted 4 May 2026cs.LG cs.AI cs.CRcited by 0

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Safety fine-tuning of language models typically requires a curated adversarial dataset. We take a different approach: score each candidate prompt's difficulty by how often the target model's own rollouts are judged harmful, then fine-tune on the hardest prompts paired with the model's own non-jailbroken rollouts. On Llama-3-8B-Instruct and Llama-3.2-3B-Instruct, this approach cuts the WildJailbreak attack success rate from 11.5% and 20.1% down to 1-3%, but pushes refusal on jailbreak-shaped benign prompts from 14-22% to 74-94%. Interleaving the same hard prompts 1:1 with adversarially-framed benign prompts (prompts that look like jailbreaks but have benign intent) cuts that refusal back down to 30-51% on 8B and 52-72% on 3B, at a cost of 2-6 percentage points of attack success rate. Within the mixed regime, training on the hardest half of the eligible pool rather than a random half cuts the remaining ASR by 35-50% (about 3 percentage points) on both models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 9 of its 13 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

9 of 13 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Consistency Training Helps Stop Sycophancy and Jailbreaks31 Oct
  2. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models26 Jun
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  4. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 5

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 579
  3. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  4. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  5. 2024Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thoughtcited by 39

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
  2. 2026Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks
  3. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Study
  4. 2024Unleashing the Unseen: Harnessing Benign Datasets for Jailbreaking Large Language Models
  5. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety
  6. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research
  7. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
  8. 2026When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Models