mlsanityData to 7 Oct 2026

Improved Generation of Adversarial Examples Against Safety-aligned LLMs

arXiv 2405.20778 ↗First posted 28 May 2024cs.CR cs.LGcited by 17

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Adversarial prompts generated using gradient-based methods exhibit outstanding performance in performing automatic jailbreak attacks against safety-aligned LLMs. Nevertheless, due to the discrete nature of texts, the input gradient of LLMs struggles to precisely reflect the magnitude of loss change that results from token replacements in the prompt, leading to limited attack success rates against safety-aligned LLMs, even in the white-box setting. In this paper, we explore a new perspective on this problem, suggesting that it can be alleviated by leveraging innovations inspired in transfer-based attacks that were originally proposed for attacking black-box image classification models. For the first time, we appropriate the ideologies of effective methods among these transfer-based attacks, i.e., Skip Gradient Method and Intermediate Level Attack, into gradient-based adversarial prompt generation and achieve significant performance gains without introducing obvious computational cost. Meanwhile, by discussing mechanisms behind the gains, new insights are drawn, and proper combinations of these methods are also developed. Our empirical results show that 87% of the query-specific adversarial suffixes generated by the developed combination can induce Llama-2-7B-Chat to produce the output that exactly matches the target string on AdvBench. This match rate is 33% higher than that of a very strong baseline known as GCG, demonstrating advanced discrete optimization for adversarial prompt generation against LLMs. In addition, without introducing obvious cost, the combination achieves >30% absolute increase in attack success rates compared with GCG when generating both query-specific (38% -> 68%) and universal adversarial prompts (26.68% -> 60.32%) for attacking the Llama-2-7B-Chat model on AdvBench. Code at: https://github.com/qizhangli/Gradient-based-Jailbreak-Attacks.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 48 of its 61 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

48 of 61 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  3. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  5. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  8. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2022Mass-Editing Memory in a Transformercited by 976
  11. 2016Adversarial Machine Learning at Scalecited by 934
  12. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
28 more
  1. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  2. 2016Delving into Transferable Adversarial Examples and Black-box Attackscited by 530
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 439
  5. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  6. 2023Are aligned neural networks adversarially aligned?cited by 305
  7. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  8. 2019Simple Black-box Adversarial Attackscited by 249
  9. 2023Pretraining Language Models with Human Preferencescited by 245
  10. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  11. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  12. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  13. 2019Enhancing Adversarial Example Transferability with an Intermediate Level Attackcited by 139
  14. 2021Feature Importance-aware Transferable Adversarial Attackscited by 125
  15. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  16. 2022Improving Adversarial Transferability via Neuron Attribution-Based Attackscited by 92
  17. 2021Towards Transferable Adversarial Attacks on Vision Transformerscited by 80
  18. 2020Backpropagating Linearly Improves Transferability of Adversarial Examplescited by 79
  19. 2021On Improving Adversarial Transferability of Vision Transformerscited by 70
  20. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 46
  21. 2022LGV: Boosting Adversarial Example Transferability from Large Geometric Vicinitycited by 39
  22. 2020Yet Another Intermediate-Level Attackcited by 35
  23. 2023Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examplescited by 32
  24. 2023Improving Adversarial Transferability via Intermediate-level Perturbation Decaycited by 29
  25. 2022An Intermediate-level Attack Framework on The Basis of Linear Regressioncited by 9
  26. 2023Towards Evaluating Transfer-based Attacks Systematically, Practically, and Fairlycited by 6
  27. 2023Improving Transferability of Adversarial Examples via Bayesian Attackscited by 3
  28. 2021A Theoretical View of Linear Backpropagation and Its Convergencecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 6.

17 papers.

Sort:

By area:

  1. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  2. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  3. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 10
  4. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  5. 2025SAGE: A Generic Framework for LLM Safety Evaluationcited by 7
  6. 2024Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translationcited by 6
  7. 2025Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Educationcited by 5
  8. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attackscited by 4
  9. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  10. 2025The Resurgence of GCG Adversarial Attacks on Large Language Modelscited by 2
  11. 2026Unsafe by Reciprocity: How Generation-Understanding Coupling Undermines Safety in Unified Multimodal Modelscited by 2
  12. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
5 more
  1. 2025One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMscited by 1
  2. 2026Efficient Refusal Ablation in LLM through Optimal Transportcited by 1
  3. 2026Amnesia: Adversarial Semantic Layer Specific Activation Steering in Large Language Modelscited by 0
  4. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoringcited by 0
  5. 2026A Single Suffix to Break Them All: Basin-Aware Jailbreaks for Merged Model Familiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The TIP of the Iceberg: Revealing a Hidden Class of Task-in-Prompt Adversarial Attacks on LLMs
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models
  3. 2025Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs
  4. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs
  5. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
  6. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compression
  7. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models
  8. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey