mlsanityData to 7 Oct 2026

Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs

arXiv 2406.09324 ↗First posted 13 Jun 2024cs.CR cs.AI cs.CLcited by 47

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Although Large Language Models (LLMs) have demonstrated significant capabilities in executing complex tasks in a zero-shot manner, they are susceptible to jailbreak attacks and can be manipulated to produce harmful outputs. Recently, a growing body of research has categorized jailbreak attacks into token-level and prompt-level attacks. However, previous work primarily overlooks the diverse key factors of jailbreak attacks, with most studies concentrating on LLM vulnerabilities and lacking exploration of defense-enhanced LLMs. To address these issues, we introduced JailTrickBench to evaluate the impact of various attack settings on LLM performance and provide a baseline for jailbreak attacks, encouraging the adoption of a standardized evaluation framework. Specifically, we evaluate the eight key factors of implementing jailbreak attacks on LLMs from both target-level and attack-level perspectives. We further conduct seven representative jailbreak attacks on six defense methods across two widely used datasets, encompassing approximately 354 experiments with about 55,000 GPU hours on A800-80G. Our experimental results highlight the need for standardized benchmarking to evaluate these attacks on defense-enhanced LLMs. Our code is available at https://github.com/usail-hkust/JailTrickBench.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 72 of its 87 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

72 of 87 references.

Closest ancestors · 46

Same direction, newest first

  1. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs7 Jun
  2. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses3 Jun
  3. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Models31 May
  4. 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters30 May
  5. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks30 May
  6. 2024Improved Generation of Adversarial Examples Against Safety-aligned LLMs28 May
  7. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing28 May
  8. 2024Efficient Adversarial Training in LLMs with Continuous Attacks24 May
  9. 2024GPT-4 Jailbreaks Itself with Near-Perfect Success Using Self-Explanation21 May
  10. 2024Protecting Your LLMs with Information Bottleneck22 Apr
  11. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  12. 2024JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models12 Apr
34 more
  1. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
  2. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge8 Apr
  3. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  4. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  5. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack2 Apr
  6. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  7. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content19 Mar
  8. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks2 Mar
  9. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes1 Mar
  10. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction28 Feb
  11. 2024Defending LLMs against Jailbreaking Attacks via Backtranslation26 Feb
  12. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing25 Feb
  13. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails24 Feb
  14. 2024Is the System Message Really Important to Jailbreaks in Large Language Models?20 Feb
  15. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  16. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability13 Feb
  17. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs8 Feb
  18. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  19. 2024Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning19 Jan
  20. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  21. 2023Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak7 Dec
  22. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  23. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  24. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  25. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  26. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  27. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  28. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  29. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  30. 2023Detecting Language Model Attacks with Perplexity27 Aug
  31. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  32. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  33. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models17 Jul
  34. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 26

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  8. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  9. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  10. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  11. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  12. 2023Large Language Model Unlearningcited by 288
14 more
  1. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  2. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  3. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  4. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  5. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  6. 2024Chain of Attack: a Semantic-Driven Contextual Multi-Turn attacker for LLMcited by 73
  7. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  8. 2023LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?cited by 47
  9. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curationcited by 43
  10. 2024Prompt Stealing Attacks Against Large Language Modelscited by 39
  11. 2024Model Extrapolation Expedites Alignmentcited by 37
  12. 2022Practical Adversarial Attacks on Spatiotemporal Traffic Forecasting Modelscited by 28
  13. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignmentcited by 16
  14. 2024BELLS: A Framework Towards Future Proof Benchmarks for the Evaluation of LLM Safeguardscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 19; 2026: 25.

47 papers.

Sort:

By area:

  1. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  2. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  3. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  4. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 36
  5. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysiscited by 27
  6. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  7. 2026MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attackscited by 18
  8. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 13
  9. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMscited by 12
  10. 2025OmniSafeBench-MM: A Unified Benchmark and Toolbox for Multimodal Jailbreak Attack-Defense Evaluationcited by 10
  11. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 10
  12. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 9
35 more
  1. 2025LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMscited by 7
  2. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  3. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safetycited by 7
  4. 2026What Matters For Safety Alignment?cited by 6
  5. 2026HoneyTrap: Deceiving Large Language Model Attackers to Honeypot Traps with Resilient Multi-Agent Defensecited by 6
  6. 2026JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attackscited by 6
  7. 2025MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?cited by 5
  8. 2026Now You Hear Me: Audio Narrative Attacks Against Large Audio-Language Modelscited by 5
  9. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attackscited by 4
  10. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  11. 2024Preventing Jailbreak Prompts as Malicious Tools for Cybercriminals: A Cyber Defense Perspectivecited by 3
  12. 2025STShield: Single-Token Sentinel for Real-Time Jailbreak Detection in Large Language Modelscited by 3
  13. 2026The Echo Chamber Multi-Turn LLM Jailbreakcited by 3
  14. 2026MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attackscited by 3
  15. 2025AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokenscited by 2
  16. 2026Multi-turn Jailbreaking Attack in Multi-Modal Large Language Modelscited by 2
  17. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 2
  18. 2025Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defensescited by 1
  19. 2025MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Securitycited by 1
  20. 2025RAGs to Riches: RAG-like Few-shot Learning for Large Language Model Role-playingcited by 1
  21. 2026TrojanPraise: Jailbreak LLMs via Benign Fine-Tuningcited by 1
  22. 2026Toward Reliable, Safe, and Secure LLMs for Scientific Applicationscited by 1
  23. 2026Seed Hijacking of LLM Sampling and Quantum Random Number Defensecited by 1
  24. 2026Voices Across Registers: Corpus-Conditioned Vernacular Jailbreaks against Aligned LLMs via Fanfiction Subgenrescited by 1
  25. 2026Risk Under Pressure: Compute-Aware Evaluation of Adversarial Robustness in Language Modelscited by 1
  26. 2025MuTRAP: Multi-trigger Trojans Attacking Robot Task Planning Systemscited by 0
  27. 2026Silencing the Guardrails: Inference-Time Jailbreaking via Dynamic Contextual Representation Ablationcited by 0
  28. 2026You Snooze, You Lose: Automatic Safety Alignment Restoration through Neural Weight Translationcited by 0
  29. 2026Single-Configuration Attack Success Rate Is Not Enough: Jailbreak Evaluations Should Report Distributional Attack Successcited by 0
  30. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoringcited by 0
  31. 2026Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Turn Attackscited by 0
  32. 2026When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMscited by 0
  33. 2026The Fragility of Jailbreak Robustness Across Operational Statescited by 0
  34. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  35. 2026VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformationcited by 0