mlsanityData to 14 Oct 2026

Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks

arXiv 2508.20038 ↗First posted 27 Aug 2025cs.CLcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Despite advances in improving large language model (LLM) to refuse to answer malicious instructions, widely used LLMs remain vulnerable to jailbreak attacks where attackers generate instructions with distributions differing from safety alignment corpora. New attacks expose LLMs' inability to recognize unseen malicious instructions, highlighting a critical distributional mismatch between training data and real-world attacks that forces developers into reactive patching cycles. To tackle this challenge, we propose IMAGINE, a synthesis framework that leverages embedding space distribution analysis to generate jailbreak-like instructions. This approach effectively fills the distributional gap between authentic jailbreak patterns and safety alignment corpora. IMAGINE follows an iterative optimization process that dynamically evolves text generation distributions across iterations, thereby augmenting the coverage of safety alignment data distributions through synthesized data examples. Based on the safety-aligned corpus enhanced through IMAGINE, our framework demonstrates significant decreases in attack success rate on Qwen2.5, Llama3.1, and Llama3.2 without compromising their utility.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 42 references that are in the map. First posted on arXiv 27 Aug 2025.

Builds on

34 of 42 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring11 Jun
  2. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference20 Jun
  3. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  4. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States9 Jun
  5. 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vector18 Apr
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  8. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep

Other references · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2020Supervised Contrastive Learningcited by 2,829
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  12. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
14 more
  1. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  2. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  3. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  4. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  5. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  6. 2024Direct Preference Optimization with an Offsetcited by 122
  7. 2023Progressive Open Space Expansion for Open-Set Model Attributioncited by 32
  8. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 26
  9. 2025Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Mergingcited by 23
  10. 2025LLM-Generated Fake News Induces Truth Decay in News Ecosystem: A Case Study on Neural News Recommendationcited by 19
  11. 2025Following the Autoregressive Nature of LLM Embeddings via Compression and Alignmentcited by 14
  12. 2024Automated Progressive Red Teamingcited by 13
  13. 2023ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preferencecited by 12
  14. 2024Vision-fused Attack: Advancing Aggressive and Stealthy Adversarial Text against Neural Machine Translationcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025LLM Latent Reasoning as Chain of Superpositioncited by 39

Similar, not cited

Close in wording, no citation either way

  1. 2025Stand on The Shoulders of Giants: Building JailExpert from Previous Attack Experience
  2. 2025Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expression
  3. 2024Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis
  4. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
  5. 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy
  6. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
  7. 2023Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak
  8. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures