mlsanityData to 7 Oct 2026

Tree of Attacks: Jailbreaking Black-Box LLMs Automatically

arXiv 2312.02119 ↗First posted 4 Dec 2023cs.LG cs.AI cs.CLcited by 619

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

While Large Language Models (LLMs) display versatile functionality, they continue to generate harmful, biased, and toxic content, as demonstrated by the prevalence of human-designed jailbreaks. In this work, we present Tree of Attacks with Pruning (TAP), an automated method for generating jailbreaks that only requires black-box access to the target LLM. TAP utilizes an attacker LLM to iteratively refine candidate (attack) prompts until one of the refined prompts jailbreaks the target. In addition, before sending prompts to the target, TAP assesses them and prunes the ones unlikely to result in jailbreaks, reducing the number of queries sent to the target LLM. In empirical evaluations, we observe that TAP generates prompts that jailbreak state-of-the-art LLMs (including GPT4-Turbo and GPT4o) for more than 80% of the prompts. This significantly improves upon the previous state-of-the-art black-box methods for generating jailbreaks while using a smaller number of queries than them. Furthermore, TAP is also capable of jailbreaking LLMs protected by state-of-the-art guardrails, e.g., LlamaGuard.

Key facts. Cited by 619 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 40 of its 56 references that are in the map. First posted on arXiv 4 Dec 2023.

Builds on

40 of 56 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts15 Nov
  2. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  5. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  6. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  8. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  9. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 31

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  10. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  12. 2020Aligning AI With Shared Human Valuescited by 706
19 more
  1. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  2. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  3. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  4. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  5. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  6. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  7. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  8. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  9. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  10. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  11. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  12. 2023Gender bias and stereotypes in Large Language Modelscited by 185
  13. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  14. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  15. 2020It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbationscited by 64
  16. 2022"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicecited by 55
  17. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  18. 2023LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Modelcited by 21
  19. 2022Debiased Large Language Models Still Associate Muslims with Uniquely Violent Actscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 140; 2025: 212; 2026: 266.

619 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  3. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  4. 2024A StrongREJECT for Empty Jailbreakscited by 352
  5. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  6. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  7. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  8. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  9. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
  10. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  11. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  12. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
38 more
  1. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  2. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  3. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  4. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  5. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  6. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 126
  7. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 126
  8. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  9. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  10. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  11. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  12. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  13. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  14. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  15. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  16. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  17. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  18. 2024Attacking Large Language Models with Projected Gradient Descentcited by 91
  19. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  20. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  21. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  22. 2024Safeguarding Large Language Models: A Surveycited by 73
  23. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  24. 2024Best-of-N Jailbreakingcited by 71
  25. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothingcited by 68
  26. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  27. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  28. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 63
  29. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapescited by 63
  30. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  31. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 61
  32. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 57
  33. 2024EasyJailbreak: A Unified Framework for Jailbreaking Large Language Modelscited by 57
  34. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  35. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agentscited by 57
  36. 2024LLMmap: Fingerprinting For Large Language Modelscited by 56
  37. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  38. 2024Query-Based Adversarial Prompt Generationcited by 55

50 most cited of 619 All 619 (JSON)