mlsanityData to 14 Oct 2026

Efficient Adversarial Training in LLMs with Continuous Attacks

arXiv 2405.15589 ↗First posted 24 May 2024cs.LG cs.CRcited by 113

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models (LLMs) are vulnerable to adversarial attacks that can bypass their safety guardrails. In many domains, adversarial training has proven to be one of the most promising methods to reliably improve robustness against such attacks. Yet, in the context of LLMs, current methods for adversarial training are hindered by the high computational costs required to perform discrete adversarial attacks at each training iteration. We address this problem by instead calculating adversarial attacks in the continuous embedding space of the LLM, which is orders of magnitudes more efficient. We propose a fast adversarial training algorithm (C-AdvUL) composed of two losses: the first makes the model robust on continuous embedding attacks computed on an adversarial behaviour dataset; the second ensures the usefulness of the final model by fine-tuning on utility data. Moreover, we introduce C-AdvIPO, an adversarial variant of IPO that does not require utility data for adversarially robust alignment. Our empirical evaluation on five models from different families (Gemma, Phi3, Mistral, Zephyr, Llama2) and at different scales (2B, 3.8B, 7B) shows that both algorithms substantially enhance LLM robustness against discrete attacks (GCG, AutoDAN, PAIR), while maintaining utility. Our results demonstrate that robustness to continuous perturbations can extrapolate to discrete threat models. Thereby, we present a path toward scalable adversarial training algorithms for robustly aligning LLMs.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 38 of its 46 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

38 of 46 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  3. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  4. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space14 Feb
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  6. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  7. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  8. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 29

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  10. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
17 more
  1. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  3. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  4. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  5. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  6. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 264
  7. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  8. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  9. 2020Adversarial Training for Large Neural Language Modelscited by 111
  10. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 102
  11. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 88
  12. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  13. 2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
  14. 2023Scaling Laws for Adversarial Attacks on Language Model Activationscited by 19
  15. 2021Identifying Untrustworthy Predictions in Neural Networks by Geometric Gradient Analysiscited by 13
  16. 2023Raising the Bar for Certified Adversarial Robustness with Diffusion Modelscited by 7
  17. 2024In-Context Learning Can Re-learn Forbidden Taskscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 16; 2025: 58; 2026: 39.

113 papers.

Sort:

By area:

  1. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  2. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  3. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  4. 2025Memory Injection Attacks on LLM Agents via Query-Only Interactioncited by 144
  5. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 115
  6. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 64
  7. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 63
  8. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 54
  9. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 51
  10. 2025Rethinking Reflection in Pre-Trainingcited by 43
  11. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  12. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 39
38 more
  1. 2025Adversarial Reasoning at Jailbreaking Timecited by 29
  2. 2024Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Modelscited by 27
  3. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 26
  4. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 24
  5. 2024Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustnesscited by 22
  6. 2024LLM-based Multi-Agent Systems: Techniques and Business Perspectivescited by 22
  7. 2024The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?cited by 22
  8. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  9. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21
  10. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 21
  11. 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustnesscited by 20
  12. 2024Endless Jailbreaks with Bijection Learningcited by 19
  13. 2025LLM-Safety Evaluations Lack Robustnesscited by 18
  14. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 16
  15. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 14
  16. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  17. 2025Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Trainingcited by 12
  18. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  19. 2025MixAT: Combining Continuous and Discrete Adversarial Training for LLMscited by 11
  20. 2025Robust Answers, Fragile Logic: Probing the Decoupling Hypothesis in LLM Reasoningcited by 11
  21. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 11
  22. 2025In-Context Defense in Computer Agents: An Empirical Studycited by 10
  23. 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Modelcited by 10
  24. 2025Jailbreaking to Jailbreakcited by 9
  25. 2025Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Modelscited by 9
  26. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectivescited by 8
  27. 2025Context Reasoner: Incentivizing Reasoning Capability for Contextualized Privacy and Safety Compliance via Reinforcement Learningcited by 8
  28. 2025Improving Large Language Model Safety with Contrastive Representation Learningcited by 8
  29. 2025Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal Directioncited by 8
  30. 2025Retrieval-Augmented Purifier for Robust LLM-Empowered Recommendationcited by 7
  31. 2025Adversarial Preference Learning for Robust LLM Alignmentcited by 7
  32. 2025Should LLM Safety Be More Than Refusing Harmful Instructions?cited by 7
  33. 2025Adversarial D'ej`a Vu: Jailbreak Dictionary Learning for Stronger Generalization to Unseen Attackscited by 7
  34. 2025Diffusion LLMs are Natural Adversaries for any LLMcited by 7
  35. 2025TeleAI-Safety: A comprehensive LLM jailbreaking benchmark towards attacks, defenses, and evaluationscited by 7
  36. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 6
  37. 2025Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocolcited by 6
  38. 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replaycited by 6

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Adversarial Attacks on Large Language Models Using Regularized Relaxation
  2. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research
  3. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  4. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models
  5. 2025Adversarial Attack on Large Language Models using Exponentiated Gradient Descent
  6. 2024Large Language Model Sentinel: LLM Agent for Adversarial Purification
  7. 2025Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuning
  8. 2025Black-box Optimization of LLM Outputs by Asking for Directions