mlsanityData to 13 Oct 2026

Baseline Defenses for Adversarial Attacks Against Aligned Language Models

arXiv 2309.00614 ↗First posted 1 Sep 2023cs.LG cs.CL cs.CRcited by 647

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

As Large Language Models quickly become ubiquitous, it becomes critical to understand their security vulnerabilities. Recent work shows that text optimizers can produce jailbreaking prompts that bypass moderation and alignment. Drawing from the rich body of work on adversarial machine learning, we approach these attacks with three questions: What threat models are practically useful in this domain? How do baseline defense techniques perform in this new domain? How does LLM security differ from computer vision? We evaluate several baseline defense strategies against leading adversarial attacks on LLMs, discussing the various settings in which each is feasible and effective. Particularly, we look at three types of defenses: detection (perplexity based), input preprocessing (paraphrase and retokenization), and adversarial training. We discuss white-box and gray-box settings and discuss the robustness-performance trade-off for each of the defenses considered. We find that the weakness of existing discrete optimizers for text, combined with the relatively high costs of optimization, makes standard adaptive attacks more challenging for LLMs. Future research will be needed to uncover whether more powerful optimizers can be developed, or whether the strength of filtering and preprocessing defenses is greater in the LLMs domain than it has been in computer vision.

Key facts. Cited by 647 papers in the map (data through 13 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 61 of its 75 references that are in the map. First posted on arXiv 1 Sep 2023.

Builds on

61 of 75 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  2. 2023Are aligned neural networks adversarially aligned?26 Jun
  3. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 57

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,336
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,379
  5. 2013Intriguing properties of neural networkscited by 5,137
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,388
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  10. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,417
  11. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,265
  12. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 817
38 more
  1. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 760
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 745
  3. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 718
  4. 2019Adversarial Training for Free!cited by 641
  5. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 637
  6. 2020On Adaptive Attacks to Adversarial Example Defensescited by 554
  7. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 545
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 529
  9. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 490
  10. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 485
  11. 2022Diffusion Models for Adversarial Purificationcited by 472
  12. 2019On Evaluating Adversarial Robustnesscited by 447
  13. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  14. 2018Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Modelscited by 362
  15. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 351
  16. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 344
  17. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 335
  18. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  19. 2021Unsolved Problems in ML Safetycited by 291
  20. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 277
  21. 2017MagNet: a Two-Pronged Defense against Adversarial Examplescited by 269
  22. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 263
  23. 2023Pretraining Language Models with Human Preferencescited by 246
  24. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  25. 2017On Detecting Adversarial Perturbationscited by 232
  26. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 224
  27. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 194
  28. 2021Data Augmentation Can Improve Robustnesscited by 193
  29. 2023On the Reliability of Watermarks for Large Language Modelscited by 187
  30. 2017On the (Statistical) Detection of Adversarial Examplescited by 180
  31. 2019Making an Invisibility Cloak: Real World Adversarial Attacks on Object Detectorscited by 155
  32. 2014Towards Deep Neural Network Architectures Robust to Adversarial Examplescited by 146
  33. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 114
  34. 2018Semantic Adversarial Examplescited by 86
  35. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 82
  36. 2021Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Themcited by 47
  37. 2020TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLPcited by 45
  38. 2017Enhancing Robustness of Machine Learning Systems via Data Transformationscited by 38

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 31; 2024: 215; 2025: 240; 2026: 161.

647 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,405
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,404
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 777
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 565
  5. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 531
  6. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 483
  7. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 464
  8. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 448
  9. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 429
  10. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  11. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 362
  12. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 360
38 more
  1. 2024Improving Alignment and Robustness with Circuit Breakerscited by 318
  2. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 265
  3. 2023Certifying LLM Safety against Adversarial Promptingcited by 263
  4. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  5. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 245
  6. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 229
  7. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  8. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 214
  9. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 192
  10. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 185
  11. 2023Can LLM-Generated Misinformation Be Detected?cited by 176
  12. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 173
  13. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 171
  14. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 169
  15. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  16. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  17. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 156
  18. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 152
  19. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 145
  20. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 142
  21. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 142
  22. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  23. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 128
  24. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 127
  25. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  26. 2025LlamaFirewall: An open source guardrail system for building secure AI agentscited by 123
  27. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 123
  28. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 120
  29. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 120
  30. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 119
  31. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 118
  32. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 118
  33. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 112
  34. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 109
  35. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  36. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 106
  37. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 106
  38. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 105

50 most cited of 647 All 647 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  2. 2025ShieldLearner: A New Paradigm for Jailbreak Attack Defense in LLMs
  3. 2026Adversarial Reframing: A Framework for Targeted Generation in Language Models
  4. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs
  5. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectives
  6. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective
  7. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence
  8. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Research