mlsanityData to 14 Oct 2026

JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models

arXiv 2406.09321 ↗First posted 13 Jun 2024cs.CR cs.AI cs.CLcited by 31

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Jailbreak attacks induce Large Language Models (LLMs) to generate harmful responses, posing severe misuse threats. Though research on jailbreak attacks and defenses is emerging, there is no consensus on evaluating jailbreaks, i.e., the methods to assess the harmfulness of an LLM's response are varied. Each approach has its own set of strengths and weaknesses, impacting their alignment with human values, as well as the time and financial cost. This diversity challenges researchers in choosing suitable evaluation methods and comparing different attacks and defenses. In this paper, we conduct a comprehensive analysis of jailbreak evaluation methodologies, drawing from nearly 90 jailbreak research published between May 2023 and April 2024. Our study introduces a systematic taxonomy of jailbreak evaluators, offering indepth insights into their strengths and weaknesses, along with the current status of their adaptation. To aid further research, we propose JailbreakEval, a toolkit for evaluating jailbreak attempts. JailbreakEval includes various evaluators out-of-the-box, enabling users to obtain results with a single command or customized evaluation workflows. In summary, we regard JailbreakEval to be a catalyst that simplifies the evaluation process in jailbreak research and fosters an inclusive standard for jailbreak evaluation within the community.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 89 of its 95 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

89 of 95 references.

Closest ancestors · 58

Same direction, newest first

  1. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge8 Apr
  2. 2024Red Teaming GPT-4V: Are GPT-4V Safe Against Uni/Multi-Modal Jailbreak Attacks?4 Apr
  3. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  4. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  5. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack2 Apr
  6. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  7. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models26 Mar
  8. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content19 Mar
  9. 2024EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models18 Mar
  10. 2024Distract Large Language Models for Automatic Jailbreak Attack13 Mar
  11. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks2 Mar
  12. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction28 Feb
38 more
  1. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models26 Feb
  2. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers25 Feb
  3. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing25 Feb
  4. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails24 Feb
  5. 2024Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs21 Feb
  6. 2024Coercing LLMs to do and reveal (almost) anything21 Feb
  7. 2024Is the System Message Really Important to Jailbreaks in Large Language Models?20 Feb
  8. 2024Query-Based Adversarial Prompt Generation19 Feb
  9. 2024A StrongREJECT for Empty Jailbreaks15 Feb
  10. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues14 Feb
  11. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
  12. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability13 Feb
  13. 2024Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning13 Feb
  14. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs8 Feb
  15. 2024GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Models5 Feb
  16. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  17. 2024Weak-to-Strong Jailbreaking on Large Language Models30 Jan
  18. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Models30 Jan
  19. 2024Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning19 Jan
  20. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks18 Jan
  21. 2024AttackEval: How to Evaluate the Effectiveness of Jailbreak Attacking on Large Language Models17 Jan
  22. 2024Intention Analysis Makes LLMs A Good Jailbreak Defender12 Jan
  23. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  24. 2023Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak7 Dec
  25. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  26. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts15 Nov
  27. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
  28. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  29. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  30. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  31. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  32. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  33. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  34. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  35. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  36. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  37. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  38. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep

50 newest of 58 All 58 (JSON)

Other references · 31

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  7. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  8. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  9. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  10. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  11. 2023AI Alignment: A Comprehensive Surveycited by 263
  12. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
19 more
  1. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  2. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  3. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  4. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 163
  5. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 136
  6. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  7. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 125
  8. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 116
  9. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 80
  10. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 76
  11. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 50
  12. 2023Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experimentcited by 36
  13. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  14. 2023Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMscited by 27
  15. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 27
  16. 2023Intelligent Virtual Assistants with LLM-based Process Automationcited by 26
  17. 2023Decoding the Threat Landscape : ChatGPT, FraudGPT, and WormGPT in Social Engineering Attackscited by 23
  18. 2024Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Modelscited by 20
  19. 2024Fine-Tuning, Quantization, and LLMs: Navigating Unintended Outcomescited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 19; 2026: 8.

31 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  2. 2024SafeBench: A Safety Evaluation Framework for Multimodal Large Language Modelscited by 69
  3. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  4. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 47
  5. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  6. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 31
  7. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  8. 2025HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessmentcited by 15
  9. 2025Adversarial Tokenizationcited by 11
  10. 2025Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMscited by 11
  11. 2025Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcarecited by 10
  12. 2025Amplified Vulnerabilities: Structured Jailbreak Attacks on LLM-based Multi-Agent Debatecited by 10
19 more
  1. 2025Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerabilitycited by 10
  2. 2025HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Modelcited by 9
  3. 2025SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requestscited by 5
  4. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 5
  5. 2025Toward Principled LLM Safety Testing: Solving the Jailbreak Oracle Problemcited by 4
  6. 2025When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputscited by 4
  7. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failurescited by 3
  8. 2025PRISM: Programmatic Reasoning with Image Sequence Manipulation for LVLM Jailbreakingcited by 3
  9. 2026How Real is Your Jailbreak? Fine-grained Jailbreak Evaluation with Anchored Referencecited by 3
  10. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 3
  11. 2025ResponsibleRobotBench: Benchmarking Responsible Robot Manipulation using Multi-modal Large Language Modelscited by 2
  12. 2026Proactive Hardening of LLM Defenses with HASTEcited by 2
  13. 2026SafeDream: Safety World Model for Proactive Early Jailbreak Detectioncited by 2
  14. 2025Machine Learning for Detection and Analysis of Novel LLM Jailbreakscited by 1
  15. 2026David vs. Goliath: Verifiable Agent-to-Agent Jailbreaking via Reinforcement Learningcited by 1
  16. 2026When Scanners Lie: Evaluator Instability in LLM Red-Teamingcited by 1
  17. 2025How to Trick Your AI TA: A Systematic Study of Academic Jailbreaking in LLM Code Evaluationcited by 0
  18. 2026Escaping Alignment: A Physical Trap Model of Best-of-N Jailbreakingcited by 0
  19. 2026How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defensecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models
  2. 2024Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks
  3. 2024Rethinking How to Evaluate Language Model Jailbreak
  4. 2026JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models
  5. 2025Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detection
  6. 2026An Empirical Measurement of Jailbreaking Evaluators
  7. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs
  8. 2025Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions