mlsanityData to 7 Oct 2026

AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs

arXiv 2404.16873 ↗First posted 21 Apr 2024cs.CR cs.AI cs.CLcited by 146

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) are vulnerable to jailbreaking attacks that lead to generation of inappropriate or harmful content. Manual red-teaming requires a time-consuming search for adversarial prompts, whereas automatic adversarial prompt generation often leads to semantically meaningless attacks that do not scale well. In this paper, we present a novel method that uses another LLM, called AdvPrompter, to generate human-readable adversarial prompts in seconds. AdvPrompter, which is trained using an alternating optimization algorithm, generates suffixes that veil the input instruction without changing its meaning, such that the TargetLLM is lured to give a harmful response. Experimental results on popular open source TargetLLMs show highly competitive results on the AdvBench and HarmBench datasets, that also transfer to closed-source black-box LLMs. We also show that training on adversarial suffixes generated by AdvPrompter is a promising strategy for improving the robustness of LLMs to jailbreaking attacks.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 44 of its 58 references that are in the map. First posted on arXiv 21 Apr 2024.

Builds on

44 of 58 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
  2. 2024Fast Adversarial Attacks on Language Models In One GPU Minute23 Feb
  3. 2024A StrongREJECT for Empty Jailbreaks15 Feb
  4. 2024Weak-to-Strong Jailbreaking on Large Language Models30 Jan
  5. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  6. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  8. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  9. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  10. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  11. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
1 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  11. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  12. 2023Large Language Modelscited by 774
19 more
  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  2. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  3. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  4. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  6. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  7. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  8. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  9. 2021Learning to Optimize: A Primer and A Benchmarkcited by 206
  10. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  11. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  12. 2022Tutorial on amortized optimizationcited by 83
  13. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  14. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  15. 2023Black Box Adversarial Prompting for Foundation Modelscited by 62
  16. 2022SurCo: Learning Linear Surrogates For Combinatorial Nonlinear Optimization Problemscited by 21
  17. 2023LoFT: Local Proxy Fine-tuning For Improving Transferability Of Adversarial Attacks Against Large Language Modelcited by 21
  18. 2023Landscape Surrogate: Learning Decision Losses for Mathematical Optimization Under Partial Informationcited by 17
  19. 2023AutoPlan: Automatic Planning of Interactive Decision-Making Tasks With Large Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 48; 2025: 63; 2026: 35.

146 papers.

Sort:

By area:

  1. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  2. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  3. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  4. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  7. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  8. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  9. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  10. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  11. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  12. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 79
38 more
  1. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  2. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  3. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  4. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  5. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agentscited by 57
  6. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusalcited by 53
  7. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  8. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  9. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 45
  10. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  11. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  12. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 41
  13. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  14. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 32
  15. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  16. 2024ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templatescited by 29
  17. 2024MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialoguecited by 28
  18. 2025Adversarial Reasoning at Jailbreaking Timecited by 27
  19. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  20. 2024Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Modelscited by 24
  21. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
  22. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  23. 2024Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Datacited by 21
  24. 2024Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Modelscited by 21
  25. 2024RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Evictioncited by 21
  26. 2024GASLITEing the Retrieval: Exploring Vulnerabilities in Dense Embedding-based Searchcited by 20
  27. 2025CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answeringcited by 20
  28. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarioscited by 19
  29. 2024HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Modelscited by 17
  30. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  31. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignmentcited by 16
  32. 2024Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?cited by 15
  33. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 14
  34. 2025ceLLMate: Sandboxing Browser AI Agentscited by 14
  35. 2024Legilimens: Practical and Unified Content Moderation for Large Language Model Servicescited by 13
  36. 2024LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakagecited by 13
  37. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 13
  38. 2025GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projectioncited by 13

50 most cited of 146 All 146 (JSON)