mlsanityData to 7 Oct 2026

Open Sesame! Universal Black Box Jailbreaking of Large Language Models

arXiv 2309.01446 ↗First posted 4 Sep 2023cs.CL cs.CV cs.NEcited by 128

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models (LLMs), designed to provide helpful and safe responses, often rely on alignment techniques to align with user intent and social guidelines. Unfortunately, this alignment can be exploited by malicious actors seeking to manipulate an LLM's outputs for unintended purposes. In this paper we introduce a novel approach that employs a genetic algorithm (GA) to manipulate LLMs when model architecture and parameters are inaccessible. The GA attack works by optimizing a universal adversarial prompt that -- when combined with a user's query -- disrupts the attacked model's alignment, resulting in unintended and potentially harmful outputs. Our novel approach systematically reveals a model's limitations and vulnerabilities by uncovering instances where its responses deviate from expected behavior. Through extensive experiments we demonstrate the efficacy of our technique, thus contributing to the ongoing discussion on responsible AI development by providing a diagnostic tool for evaluating and enhancing alignment of LLMs with human intent. To our knowledge this is the first automated universal black box jailbreak attack.

Key facts. Cited by 128 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 40 of its 70 references that are in the map. First posted on arXiv 4 Sep 2023.

Builds on

40 of 70 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Detecting Language Model Attacks with Perplexity27 Aug
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  4. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  12. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
24 more
  1. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  2. 2018Robustness May Be at Odds with Accuracycited by 873
  3. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  4. 2022Galactica: A Large Language Model for Sciencecited by 736
  5. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  6. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  7. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  8. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  9. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  10. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  11. 2018Generating Natural Language Adversarial Examplescited by 422
  12. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  13. 2023Challenges and Applications of Large Language Modelscited by 251
  14. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 246
  15. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  16. 2018Certified Adversarial Robustness with Additive Noisecited by 178
  17. 2018Scaling provable adversarial defensescited by 141
  18. 2021High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspacescited by 139
  19. 2022(Certified!!) Adversarial Robustness for Free!cited by 130
  20. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  21. 2020Simple and Efficient Hard Label Black-box Adversarial Attacks in Low Query Budget Regimescited by 17
  22. 2023I See Dead People: Gray-Box Adversarial Attack on Image-To-Text Modelscited by 17
  23. 2022Foiling Explanations in Deep Neural Networkscited by 14
  24. 2022An Evolutionary, Gradient-Free, Query-Efficient, Black-Box Algorithm for Generating Adversarial Instances in Deep Networkscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 65; 2025: 39; 2026: 12.

128 papers.

Sort:

By area:

  1. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  3. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  5. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  6. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  7. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  8. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  9. 2024A StrongREJECT for Empty Jailbreakscited by 352
  10. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  11. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  12. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
38 more
  1. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  2. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  3. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  4. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  5. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  6. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  7. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  8. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  9. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  10. 2024Attacking Large Language Models with Projected Gradient Descentcited by 91
  11. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  12. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  13. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  14. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  15. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  16. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
  17. 2024Safeguarding Large Language Models: A Surveycited by 73
  18. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  19. 2024Best-of-N Jailbreakingcited by 71
  20. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  21. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothingcited by 68
  22. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  23. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  24. 2024Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoningcited by 62
  25. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 61
  26. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  27. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 46
  28. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 43
  29. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  30. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agentcited by 39
  31. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 38
  32. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  33. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  34. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulationcited by 31
  35. 2023Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attackscited by 30
  36. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attackscited by 30
  37. 2024PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetitioncited by 29
  38. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28

50 most cited of 128 All 128 (JSON)