mlsanityData to 7 Oct 2026

Certifying LLM Safety against Adversarial Prompting

arXiv 2309.02705 ↗First posted 6 Sep 2023cs.CL cs.AI cs.CRcited by 248

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models (LLMs) are vulnerable to adversarial attacks that add malicious tokens to an input prompt to bypass the safety guardrails of an LLM and cause it to produce harmful content. In this work, we introduce erase-and-check, the first framework for defending against adversarial prompts with certifiable safety guarantees. Given a prompt, our procedure erases tokens individually and inspects the resulting subsequences using a safety filter. Our safety certificate guarantees that harmful prompts are not mislabeled as safe due to an adversarial attack up to a certain size. We implement the safety filter in two ways, using Llama 2 and DistilBERT, and compare the performance of erase-and-check for the two cases. We defend against three attack modes: i) adversarial suffix, where an adversarial sequence is appended at the end of a harmful prompt; ii) adversarial insertion, where the adversarial sequence is inserted anywhere in the middle of the prompt; and iii) adversarial infusion, where adversarial tokens are inserted at arbitrary positions in the prompt, not necessarily as a contiguous block. Our experimental results demonstrate that this procedure can obtain strong certified safety guarantees on harmful prompts while maintaining good empirical performance on safe prompts. Additionally, we propose three efficient empirical defenses: i) RandEC, a randomized subsampling version of erase-and-check; ii) GreedyEC, which greedily erases tokens that maximize the softmax score of the harmful class; and iii) GradEC, which uses gradient information to optimize tokens to erase. We demonstrate their effectiveness against adversarial prompts generated by the Greedy Coordinate Gradient (GCG) attack algorithm. The code for our experiments is available at https://github.com/aounon/certified-llm-safety.

Key facts. Cited by 248 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 49 of its 57 references that are in the map. First posted on arXiv 6 Sep 2023.

Builds on

49 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  2. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  10. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  11. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
35 more
  1. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
  2. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  3. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  4. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  5. 2017Countering Adversarial Images using Input Transformationscited by 517
  6. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 462
  7. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 328
  8. 2018On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Modelscited by 322
  9. 2023Pretraining Language Models with Human Preferencescited by 245
  10. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  11. 2017On the (Statistical) Detection of Adversarial Examplescited by 180
  12. 2018Certified Adversarial Robustness with Additive Noisecited by 178
  13. 2018Semidefinite relaxations for certifying robustness to adversarial examplescited by 178
  14. 2018Stochastic Activation Pruning for Robust Adversarial Defensecited by 171
  15. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  16. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  17. 2019Achieving Verified Robustness to Symbol Substitutions via Interval Bound Propagationcited by 87
  18. 2019Functional Adversarial Attackscited by 82
  19. 2016Adversarial Examples Detection in Deep Networks with Convolutional Filter Statisticscited by 77
  20. 2020SAFER: A Structure-free Approach for Certified Robustness to Adversarial Word Substitutionscited by 72
  21. 2021Skew Orthogonal Convolutionscited by 67
  22. 2017Adversarial and Clean Data Are Not Twinscited by 52
  23. 2018Training verified learners with learned verifierscited by 50
  24. 2021CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothingcited by 41
  25. 2020Second-Order Provable Defenses against Adversarial Attackscited by 39
  26. 2021Policy Smoothing for Provably Robust Reinforcement Learningcited by 39
  27. 2021Scalable Certified Segmentation via Randomized Smoothingcited by 35
  28. 2022Detection of Word Adversarial Examples in Text Classification: Benchmark and Baseline via Robust Density Estimationcited by 32
  29. 2021LAFEAT: Piercing Through Adversarial Defenses with Latent Featurescited by 24
  30. 2022Certified Robustness Against Natural Language Attacks by Causal Interventioncited by 23
  31. 2023Certified Robustness for Large Language Models with Self-Denoisingcited by 23
  32. 2022Textual Manifold-based Defense Against Natural Language Adversarial Examplescited by 19
  33. 2021Center Smoothing: Certified Robustness for Networks with Structured Outputscited by 18
  34. 2023RS-Del: Edit Distance Robustness Certificates for Sequence Classifiers via Randomized Deletioncited by 17
  35. 2023Provable Robustness for Streaming Models with a Sliding Windowcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 92; 2025: 97; 2026: 41.

248 papers.

Sort:

By area:

  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  2. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  3. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  4. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  5. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  6. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  7. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  8. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  9. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  10. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  11. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  12. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
38 more
  1. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  2. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  3. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  4. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  5. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  6. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  7. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  8. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  9. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  10. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  11. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  12. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 95
  13. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 95
  14. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  15. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden Statescited by 92
  16. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  17. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  18. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  19. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  20. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  21. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 79
  22. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  23. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothingcited by 68
  24. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  25. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  26. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 65
  27. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  28. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 63
  29. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapescited by 63
  30. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  31. 2024Building Guardrails for Large Language Modelscited by 61
  32. 2025A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memorycited by 61
  33. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  34. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  35. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Railscited by 49
  36. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledgecited by 48
  37. 2023Can LLMs Follow Simple Rules?cited by 45
  38. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44

50 most cited of 248 All 248 (JSON)