mlsanityData to 14 Oct 2026

SoK: Evaluating Jailbreak Guardrails for Large Language Models

arXiv 2506.10597 ↗First posted 12 Jun 2025cs.CR cs.AIcited by 21

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable progress, but their deployment has exposed critical vulnerabilities, particularly to jailbreak attacks that circumvent safety alignments. Guardrails--external defense mechanisms that monitor and control LLM interactions--have emerged as a promising solution. However, the current landscape of LLM guardrails is fragmented, lacking a unified taxonomy and comprehensive evaluation framework. In this Systematization of Knowledge (SoK) paper, we present the first holistic analysis of jailbreak guardrails for LLMs. We propose a novel, multi-dimensional taxonomy that categorizes guardrails along six key dimensions, and introduce a Security-Efficiency-Utility evaluation framework to assess their practical effectiveness. Through extensive analysis and experiments, we identify the strengths and limitations of existing guardrail approaches, provide insights into optimizing their defense mechanisms, and explore their universality across attack types. Our work offers a structured foundation for future research and development, aiming to guide the principled advancement and deployment of robust LLM guardrails. The code is available at https://github.com/xunguangwang/SoK4JailbreakGuardrails.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 96 of its 119 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

96 of 119 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Imposter.AI: Adversarial Attacks with Hidden Intentions towards Aligned Large Language Models22 Jul
  2. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May
  3. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  4. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing25 Feb
  5. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts23 Jan
  6. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  8. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  9. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  10. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 85

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  11. 2022Red Teaming Language Models with Language Modelscited by 1,026
  12. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
38 more
  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  2. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  3. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  5. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  6. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  7. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  8. 2023Detecting Language Model Attacks with Perplexitycited by 431
  9. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  10. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  11. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  12. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  13. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  14. 2023Certifying LLM Safety against Adversarial Promptingcited by 266
  15. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 247
  16. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  17. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
  18. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  19. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 230
  20. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 185
  21. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 183
  22. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  23. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 172
  24. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 170
  25. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  26. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  27. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 156
  28. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  29. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 145
  30. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 136
  31. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 132
  32. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 125
  33. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 113
  34. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 110
  35. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 100
  36. 2024Aligner: Efficient Alignment by Learning to Correctcited by 98
  37. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 96
  38. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 95

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 17.

21 papers.

Sort:

By area:

  1. 2026The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanismcited by 9
  2. 2026SafeClaw-R: Towards Safe and Secure Multi-Agent Personal Assistantscited by 6
  3. 2025A2AS: Agentic AI Runtime Security and Self-Defensecited by 5
  4. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 5
  5. 2025Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variationscited by 4
  6. 2026When Child Inherits: Modeling and Exploiting Subagent Spawn in Multi-Agent Networkscited by 2
  7. 2026JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Modelscited by 2
  8. 2026Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Modelscited by 1
  9. 2026Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attackscited by 1
  10. 2026From Shield to Target: Denial-of-Service Attacks on LLM-Based Agent Guardrailscited by 1
  11. 2026SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreakscited by 1
  12. 2026CASCADE Against Jailbreaks: Combination Across Stages with Controlled Attack-Defense Evaluationcited by 1
9 more
  1. 2025SEA-SafeguardBench: Evaluating AI Safety in SEA Languages and Culturescited by 0
  2. 2026TrapSuffix: Proactive Defense Against Adversarial Suffixes in Jailbreakingcited by 0
  3. 2026AI Security Research Should Better Incentivize Defense Researchcited by 0
  4. 2026Rethinking Agent Security as a Networking Problemcited by 0
  5. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  6. 2026Overflip: Repetition-Induced Label Flips in Guardrail Modelscited by 0
  7. 2026Divide, Consult, Conquer: Capability Laundering Through Aligned LLMscited by 0
  8. 2026HE-Guardrail: A Homomorphic Guardrail Against Jailbreak Attacks for Encrypted Large Language Model Inferencecited by 0
  9. 2026Benchmarking Jailbreak Guardrails for Embodied Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
  2. 2026SoK: Robustness in Large Language Models against Jailbreak Attacks
  3. 2025Involuntary Jailbreak: On Self-Prompting Attacks
  4. 2025GuardVal: Dynamic Large Language Model Jailbreak Evaluation for Comprehensive Safety Testing
  5. 2024Benchmarking LLM Guardrails in Handling Multilingual Toxicity
  6. 2025Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systems
  7. 2025Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses
  8. 2025AdaptiveGuard: Towards Adaptive Runtime Safety for LLM-Powered Software