mlsanityData to 7 Oct 2026

Attacking Large Language Models with Projected Gradient Descent

arXiv 2402.09154 ↗First posted 14 Feb 2024cs.LGcited by 91

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Current LLM alignment methods are readily broken through specifically crafted adversarial prompts. While crafting adversarial prompts using discrete optimization is highly effective, such attacks typically use more than 100,000 LLM calls. This high computational cost makes them unsuitable for, e.g., quantitative analyses and adversarial training. To remedy this, we revisit Projected Gradient Descent (PGD) on the continuously relaxed input prompt. Although previous attempts with ordinary gradient-based attacks largely failed, we show that carefully controlling the error introduced by the continuous relaxation tremendously boosts their efficacy. Our PGD for LLMs is up to one order of magnitude faster than state-of-the-art discrete optimization to achieve the same devastating attack results.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 49 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

25 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 24

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  11. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  12. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
12 more
  1. 2023The Falcon Series of Open Language Modelscited by 526
  2. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  3. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  4. 2019Topology Attack and Defense for Graph Neural Networks: An Optimization Perspectivecited by 196
  5. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  6. 2021Robustness of Graph Neural Networks at Scalecited by 100
  7. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  8. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  9. 2021Generalization of Neural Combinatorial Solvers Through the Lens of Adversarial Robustnesscited by 35
  10. 2024Gradient-Based Language Model Red Teamingcited by 33
  11. 2023Adversarial Training for Graph Neural Networks: Pitfalls, Solutions, and New Directionscited by 28
  12. 2022TextGrad: Advancing Robustness Evaluation in NLP by Gradient-Driven Optimizationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 40; 2026: 20.

91 papers.

Sort:

By area:

  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  3. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  4. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  5. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  6. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  7. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
  8. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  9. 2024Safeguarding Large Language Models: A Surveycited by 73
  10. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  11. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  12. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
38 more
  1. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  2. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 41
  3. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  4. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  5. 2024Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Samplingcited by 33
  6. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulationcited by 31
  7. 2024Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundariescited by 26
  8. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  9. 2024GPT-4 Jailbreaks Itself with Near-Perfect Success Using Self-Explanationcited by 24
  10. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
  11. 2024Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detectioncited by 20
  12. 2025Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attackscited by 20
  13. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  14. 2024Endless Jailbreaks with Bijection Learningcited by 18
  15. 2025Diagnosing and Mitigating Modality Interference in Multimodal Large Language Modelscited by 18
  16. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMscited by 18
  17. 2025LLM-Safety Evaluations Lack Robustnesscited by 16
  18. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objectivecited by 15
  19. 2025Safety Reasoning with Guidelinescited by 14
  20. 2025Lifelong Safety Alignment for Language Modelscited by 13
  21. 2024Adversarial Attacks on Large Language Models Using Regularized Relaxationcited by 12
  22. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compressioncited by 12
  23. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  24. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 10
  25. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 10
  26. 2025Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderationcited by 9
  27. 2024Testing the Limits of Jailbreaking Defenses with the Purple Problemcited by 8
  28. 2024DiffusionAttacker: Diffusion-Driven Prompt Manipulation for LLM Jailbreakcited by 8
  29. 2025DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Promptcited by 7
  30. 2025Improving Large Language Model Safety with Contrastive Representation Learningcited by 7
  31. 2025Diffusion LLMs are Natural Adversaries for any LLMcited by 7
  32. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 7
  33. 2025Bypassing Prompt Guards in Production with Controlled-Release Promptingcited by 6
  34. 2025Same Question, Different Words: A Latent Adversarial Framework for Prompt Robustnesscited by 5
  35. 2025CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMscited by 5
  36. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 5
  37. 2024Jailbreaking LLM-Controlled Robotscited by 4
  38. 2024Plentiful Jailbreaks with String Compositionscited by 4

50 most cited of 91 All 91 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models
  2. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models
  3. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  4. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMs
  5. 2025Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models
  6. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM
  7. 2024Improved Generation of Adversarial Examples Against Safety-aligned LLMs
  8. 2026Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attacks