mlsanityData to 7 Oct 2026

JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models

arXiv 2404.01318 ↗First posted 28 Mar 2024cs.CR cs.LGcited by 494

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Jailbreak attacks cause large language models (LLMs) to generate harmful, unethical, or otherwise objectionable content. Evaluating these attacks presents a number of challenges, which the current collection of benchmarks and evaluation techniques do not adequately address. First, there is no clear standard of practice regarding jailbreaking evaluation. Second, existing works compute costs and success rates in incomparable ways. And third, numerous works are not reproducible, as they withhold adversarial prompts, involve closed-source code, or rely on evolving proprietary APIs. To address these challenges, we introduce JailbreakBench, an open-sourced benchmark with the following components: (1) an evolving repository of state-of-the-art adversarial prompts, which we refer to as jailbreak artifacts; (2) a jailbreaking dataset comprising 100 behaviors -- both original and sourced from prior work (Zou et al., 2023; Mazeika et al., 2023, 2024) -- which align with OpenAI's usage policies; (3) a standardized evaluation framework at https://github.com/JailbreakBench/jailbreakbench that includes a clearly defined threat model, system prompts, chat templates, and scoring functions; and (4) a leaderboard at https://jailbreakbench.github.io/ that tracks the performance of attacks and defenses for various LLMs. We have carefully considered the potential ethical implications of releasing this benchmark, and believe that it will be a net positive for the community.

Key facts. Cited by 494 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 47 of its 76 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

47 of 76 references.

Closest ancestors · 22

Same direction, newest first

  1. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing25 Feb
  2. 2024Query-Based Adversarial Prompt Generation19 Feb
  3. 2024A StrongREJECT for Empty Jailbreaks15 Feb
  4. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  5. 2024GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Models5 Feb
  6. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks18 Jan
  7. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  8. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  9. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  11. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  12. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
10 more
  1. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  2. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  3. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  4. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  5. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  6. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  7. 2023Detecting Language Model Attacks with Perplexity27 Aug
  8. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  10. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  12. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
13 more
  1. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  2. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  3. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  4. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  5. 2023Are aligned neural networks adversarially aligned?cited by 305
  6. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  7. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  8. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 169
  9. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  10. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  11. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 46
  12. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 46
  13. 2023PromptBench: A Unified Library for Evaluation of Large Language Modelscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 68; 2025: 179; 2026: 247.

494 papers.

Sort:

By area:

  1. 2024OpenAI o1 System Cardcited by 1,220
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  3. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  4. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  5. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  6. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  7. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  8. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  9. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  10. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  11. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  12. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
38 more
  1. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 104
  2. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  3. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  4. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
  5. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  6. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  7. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 69
  8. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  9. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 67
  10. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  11. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 61
  12. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  13. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 53
  14. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  15. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  16. 2024Persistent Pre-Training Poisoning of LLMscited by 46
  17. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Modelscited by 45
  18. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 45
  19. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  20. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreakscited by 44
  21. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  22. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  23. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 41
  24. 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characterscited by 39
  25. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 38
  26. 2026Building Production-Ready Probes For Geminicited by 38
  27. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 35
  28. 2024Understanding Jailbreak Success: A Study of Latent Space Dynamics in Large Language Modelscited by 34
  29. 2024AI Risk Categorization Decoded (AIR 2024): From Government Regulations to Corporate Policiescited by 34
  30. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  31. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  32. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  33. 2025Safety Guardrails for LLM-Enabled Robotscited by 30
  34. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  35. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 29
  36. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  37. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  38. 2025Adversarial Reasoning at Jailbreaking Timecited by 27

50 most cited of 494 All 494 (JSON)