mlsanityData to 14 Oct 2026

AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs

arXiv 2404.07921 ↗First posted 11 Apr 2024cs.CLcited by 172

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

As large language models (LLMs) become increasingly prevalent and integrated into autonomous systems, ensuring their safety is imperative. Despite significant strides toward safety alignment, recent work GCG~citepzou2023universal proposes a discrete token optimization algorithm and selects the single suffix with the lowest loss to successfully jailbreak aligned LLMs. In this work, we first discuss the drawbacks of solely picking the suffix with the lowest loss during GCG optimization for jailbreaking and uncover the missed successful suffixes during the intermediate steps. Moreover, we utilize those successful suffixes as training data to learn a generative model, named AmpleGCG, which captures the distribution of adversarial suffixes given a harmful query and enables the rapid generation of hundreds of suffixes for any harmful queries in seconds. AmpleGCG achieves near 100% attack success rate (ASR) on two aligned LLMs (Llama-2-7B-chat and Vicuna-7B), surpassing two strongest attack baselines. More interestingly, AmpleGCG also transfers seamlessly to attack different models, including closed-source LLMs, achieving a 99% ASR on the latest GPT-3.5. To summarize, our work amplifies the impact of GCG by training a generative model of adversarial suffixes that is universal to any harmful queries and transferable from attacking open-source LLMs to closed-source LLMs. In addition, it can generate 200 adversarial suffixes for one harmful query in only 4 seconds, rendering it more challenging to defend.

Key facts. Cited by 172 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 43 of its 50 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

43 of 50 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content19 Mar
  2. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding14 Feb
  3. 2024Weak-to-Strong Jailbreaking on Large Language Models30 Jan
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  5. 2023Exploiting Novel GPT-4 APIs21 Dec
  6. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  7. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  8. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  9. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  10. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  11. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  12. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
3 more
  1. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2023Mistral 7Bcited by 3,164
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  10. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  11. 2022Red Teaming Language Models with Language Modelscited by 1,026
  12. 2023Large Language Models as Optimizerscited by 864
16 more
  1. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  2. 2021GPT Understands, Toocited by 716
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  5. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  6. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
  7. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  8. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
  9. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  10. 2022Iteratively Prompt Pre-trained Language Models for Chain of Thoughtcited by 75
  11. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 68
  12. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  13. 2023How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilitiescited by 33
  14. 2023On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?cited by 26
  15. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agentscited by 22
  16. 2024Towards Unified Alignment Between Agents, Humans, and Environmentcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 54; 2025: 76; 2026: 42.

172 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  2. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 199
  3. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 156
  4. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 150
  5. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 116
  6. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  7. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 110
  8. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 91
  9. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  10. 2024On the Role of Attention Heads in Large Language Model Safetycited by 72
  11. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  12. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
38 more
  1. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 67
  2. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agentscited by 59
  3. 2024Large Language Model Safety: A Holistic Surveycited by 59
  4. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusalcited by 56
  5. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 51
  6. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 47
  7. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  8. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
  9. 2025Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategycited by 44
  10. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 39
  11. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attackscited by 37
  12. 2024Crabs: Consuming Resource via Auto-generation for LLM-DoS Attack under Black-box Settingscited by 35
  13. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulationcited by 33
  14. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 29
  15. 2025Adversarial Reasoning at Jailbreaking Timecited by 29
  16. 2024Faster-GCG: Efficient Discrete Optimization Jailbreak Attacks against Aligned Large Language Modelscited by 25
  17. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  18. 2024Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detectioncited by 23
  19. 2025Understanding and Enhancing the Transferability of Jailbreaking Attackscited by 22
  20. 2025To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Modelscited by 22
  21. 2024RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Evictioncited by 21
  22. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 21
  23. 2025LLM-Safety Evaluations Lack Robustnesscited by 18
  24. 2024Virtual Context: Enhancing Jailbreak Attacks with Special Token Injectioncited by 17
  25. 2024Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilitiescited by 17
  26. 2024PBI-Attack: Prior-Guided Bimodal Interactive Black-Box Jailbreak Attack for Toxicity Maximizationcited by 17
  27. 2024Efficient Universal Goal Hijacking with Semantics-guided Prompt Organizationcited by 16
  28. 2024Exploiting the Index Gradients for Optimization-Based Jailbreaking on Large Language Modelscited by 15
  29. 2025Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactionscited by 15
  30. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objectivecited by 15
  31. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 15
  32. 2024AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lenscited by 14
  33. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 14
  34. 2024Adversarial Attacks on Large Language Models Using Regularized Relaxationcited by 12
  35. 2024Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroringcited by 12
  36. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMscited by 12
  37. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 12
  38. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMscited by 12

50 most cited of 172 All 172 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026GAS-Leak-LLM: Genetic Algorithm-Based Suffix Optimization for Black-Box LLM Jailbreaking
  2. 2024ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings
  3. 2025CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representations
  4. 2025ForgeDAN: An Evolutionary Framework for Jailbreaking Aligned Large Language Models
  5. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks
  6. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
  7. 2026Beyond Suffixes: Token Position in GCG Adversarial Attacks on Large Language Models
  8. 2024Towards Universal and Black-Box Query-Response Only Attack on LLMs with QROA