mlsanityData to 7 Oct 2026

AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models

arXiv 2310.15140 ↗First posted 23 Oct 2023cs.CR cs.AI cs.CLcited by 102

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Safety alignment of Large Language Models (LLMs) can be compromised with manual jailbreak attacks and (automatic) adversarial attacks. Recent studies suggest that defending against these attacks is possible: adversarial attacks generate unlimited but unreadable gibberish prompts, detectable by perplexity-based filters; manual jailbreak attacks craft readable prompts, but their limited number due to the necessity of human creativity allows for easy blocking. In this paper, we show that these solutions may be too optimistic. We introduce AutoDAN, an interpretable, gradient-based adversarial attack that merges the strengths of both attack types. Guided by the dual goals of jailbreak and readability, AutoDAN optimizes and generates tokens one by one from left to right, resulting in readable prompts that bypass perplexity filters while maintaining high attack success rates. Notably, these prompts, generated from scratch using gradients, are interpretable and diverse, with emerging strategies commonly seen in manual jailbreak attacks. They also generalize to unforeseen harmful behaviors and transfer to black-box LLMs better than their unreadable counterparts when using limited training data or a single proxy model. Furthermore, we show the versatility of AutoDAN by automatically leaking system prompts using a customized objective. Our work offers a new way to red-team LLMs and understand jailbreak mechanisms via interpretability.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 59 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

48 of 59 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  2. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  3. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  5. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  7. 2023Are aligned neural networks adversarially aligned?26 Jun
  8. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 40

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  11. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
28 more
  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  2. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  3. 2023A Watermark for Large Language Modelscited by 667
  4. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  5. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  6. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  7. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  8. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  9. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  10. 2023Detecting Language Model Attacks with Perplexitycited by 411
  11. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  12. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  13. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  14. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  15. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  16. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  17. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  18. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  19. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 172
  20. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  21. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  22. 2023On the Exploitability of Instruction Tuningcited by 110
  23. 2018Towards the first adversarially robust neural network model on MNISTcited by 104
  24. 2023Deception Abilities Emerged in Large Language Modelscited by 99
  25. 2023Effective Prompt Extraction from Language Modelscited by 83
  26. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  27. 2021Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Themcited by 46
  28. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 33; 2025: 48; 2026: 21.

102 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  2. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  3. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  5. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
  6. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  7. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  8. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 63
  9. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusalcited by 53
  10. 2024Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinationscited by 51
  11. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  12. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
38 more
  1. 2025Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attackscited by 32
  2. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 32
  3. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulationcited by 31
  4. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  5. 2025Why Are Web AI Agents More Vulnerable Than Standalone LLMs? A Security Analysiscited by 27
  6. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  7. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  8. 2024RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Evictioncited by 21
  9. 2024The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and Defensecited by 20
  10. 2024GASLITEing the Retrieval: Exploring Vulnerabilities in Dense Embedding-based Searchcited by 20
  11. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarioscited by 19
  12. 2024Fine-Tuning, Quantization, and LLMs: Navigating Unintended Outcomescited by 14
  13. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 14
  14. 2024Easier Said Than Done: Unpacking Intent-Behavior Gap in Jailbreaking LLM-Based Robotscited by 14
  15. 2025Refining Input Guardrails: Enhancing LLM-as-a-Judge Efficiency Through Chain-of-Thought Fine-Tuning and Alignmentcited by 14
  16. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 14
  17. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 13
  18. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 13
  19. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMscited by 12
  20. 2024Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Modelscited by 12
  21. 2025E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Modelscited by 12
  22. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMscited by 12
  23. 2024Root Defence Strategies: Ensuring Safety of LLM at the Decoding Levelcited by 11
  24. 2025Think Twice, Generate Once: Safeguarding by Progressive Self-Reflectioncited by 11
  25. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
  26. 2024Adversarial Attacks on Large Language Models in Medicinecited by 9
  27. 2024PathSeeker: Exploring LLM Security Vulnerabilities with a Reinforcement Learning-Based Jailbreak Approachcited by 9
  28. 2025SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinationscited by 9
  29. 2024Testing the Limits of Jailbreaking Defenses with the Purple Problemcited by 8
  30. 2025Universal Zero-shot Embedding Inversioncited by 8
  31. 2025Attacker's Noise Can Manipulate Your Audio-based LLM in the Real Worldcited by 8
  32. 2025Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attackscited by 8
  33. 2025AutoRAN: Automated Hijacking of Safety Reasoning in Large Reasoning Modelscited by 7
  34. 2024Targeting the Core: A Simple and Effective Method to Attack RAG-based Agents via Direct LLM Manipulationcited by 6
  35. 2025Bypassing Prompt Guards in Production with Controlled-Release Promptingcited by 6
  36. 2024LLM-Agent-UMF: LLM-based Agent Unified Modeling Framework for Seamless Design of Multi Active/Passive Core-Agent Architecturescited by 5
  37. 2024BlackDAN: A Black-Box Multi-Objective Approach for Effective and Contextual Jailbreaking of Large Language Modelscited by 5
  38. 2025Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Modelscited by 5

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  2. 2025LatentBreak: Jailbreaking Large Language Models through Latent Space Feedback
  3. 2024Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation
  4. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  5. 2024AutoBreach: Universal and Adaptive Jailbreaking with Efficient Wordplay-Guided Optimization
  6. 2025LARGO: Latent Adversarial Reflection through Gradient Optimization for Jailbreaking LLMs
  7. 2025Universal and Transferable Adversarial Attack on Large Language Models Using Exponentiated Gradient Descent
  8. 2024Defending against Jailbreak through Early Exit Generation of Large Language Models