mlsanityData to 7 Oct 2026

AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models

arXiv 2310.04451 ↗First posted 3 Oct 2023cs.CL cs.AIcited by 738

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

The aligned Large Language Models (LLMs) are powerful language understanding and decision-making tools that are created through extensive alignment with human feedback. However, these large models remain susceptible to jailbreak attacks, where adversaries manipulate prompts to elicit malicious outputs that should not be given by aligned LLMs. Investigating jailbreak prompts can lead us to delve into the limitations of LLMs and further guide us to secure them. Unfortunately, existing jailbreak techniques suffer from either (1) scalability issues, where attacks heavily rely on manual crafting of prompts, or (2) stealthiness problems, as attacks depend on token-based algorithms to generate prompts that are often semantically meaningless, making them susceptible to detection through basic perplexity testing. In light of these challenges, we intend to answer this question: Can we develop an approach that can automatically generate stealthy jailbreak prompts? In this paper, we introduce AutoDAN, a novel jailbreak attack against aligned LLMs. AutoDAN can automatically generate stealthy jailbreak prompts by the carefully designed hierarchical genetic algorithm. Extensive evaluations demonstrate that AutoDAN not only automates the process while preserving semantic meaningfulness, but also demonstrates superior attack strength in cross-model transferability, and cross-sample universality compared with the baseline. Moreover, we also compare AutoDAN with perplexity-based defense methods and show that AutoDAN can bypass them effectively.

Key facts. Cited by 738 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 37 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

25 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  2. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2022Red Teaming Language Models with Language Modelscited by 983
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  8. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  9. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  10. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 439
  11. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 378
  12. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
9 more
  1. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  2. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  3. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  4. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  5. 2023Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigationscited by 157
  6. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  7. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  8. 2020Reevaluating Adversarial Examples in Natural Languagecited by 83
  9. 2021Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processingcited by 57

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 183; 2025: 301; 2026: 243.

738 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  3. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  5. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  6. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  8. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  9. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  10. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  11. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  12. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
38 more
  1. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  2. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  3. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  4. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  5. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  6. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  7. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
  8. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  9. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  10. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  11. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  12. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  13. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  14. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  15. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  16. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  17. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  18. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  19. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
  20. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  21. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 112
  22. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  23. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  24. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  25. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  26. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 103
  27. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  28. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  29. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  30. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  31. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 94
  32. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  33. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  34. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  35. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  36. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  37. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  38. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85

50 most cited of 738 All 738 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
  2. 2024EnJa: Ensemble Jailbreak on Large Language Models
  3. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
  4. 2024SeqAR: Jailbreak LLMs with Sequential Auto-Generated Characters
  5. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  6. 2025Jailbreak Mimicry: Automated Discovery of Narrative-Based Jailbreaks for Large Language Models
  7. 2024Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring
  8. 2026Exploring and Developing a Pre-Model Safeguard with Draft Models