mlsanityData to 7 Oct 2026

SAFEGuard: Detect Optimization-Based Jailbreak Attacks Through Harmful Semantic Analysis and Fluency Measurement

arXiv 2609.05850 ↗First posted 5 Sep 2026cs.LG cs.AI cs.CRcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Despite the significant efforts devoted to aligning large language models (LLMs) with human values and ensuring safe deployment, recent work has revealed that LLMs remain vulnerable to adversarial jailbreak attacks that can bypass safety guardrails and elicit harmful responses. Many defense methods are proposed to detect jailbreaks but they are limited in their effectiveness to counter wide-range optimization-based jailbreak mechanisms that can yield highly fluency-optimized or harmful semantic obfuscated prompts. To tackle this challenge, we propose a unified detection framework SAFEGuard which incorporates a hybrid fluency measurement based on cross-layer distribution distance and perplexity, and the analysis of harmful semantics through gradient matching. Our method is grounded in a paramount observation: high fluency prompts maintain their malicious intention close to harmful prompts while harmful semantic obfuscated prompts often inject gibberish token sequences. Our evaluation demonstrates that SAFEGuard consistently outperforms state-of-the-art baselines and achieves significant improvement in accuracy across different optimization-based jailbreaks. This underscores the effectiveness of SAFEGuard against evolving jailbreak attacks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 38 references that are in the map. First posted on arXiv 5 Sep 2026.

Builds on

30 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 27

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  7. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  8. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  9. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  10. 2023Detecting Language Model Attacks with Perplexitycited by 411
  11. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  12. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
15 more
  1. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  2. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  3. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  4. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  5. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  6. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 87
  7. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  8. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  9. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  10. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapescited by 63
  11. 2024Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysiscited by 61
  12. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  13. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 45
  14. 2022How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspectivecited by 39
  15. 2025Understanding and Enhancing the Transferability of Jailbreaking Attackscited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
  2. 2026Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Models
  3. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  4. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
  5. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
  6. 2025Defense-to-Attack: Bypassing Weak Defenses Enables Stronger Jailbreaks in Vision-Language Models
  7. 2025SafeLLM: Unlearning Harmful Outputs from Large Language Models against Jailbreak Attacks
  8. 2026Circuit Discovery Helps Detect LLM Jailbreaking: A Mechanistic Interpretability Study