mlsanityData to 7 Oct 2026

Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks

arXiv 2401.17263 ↗First posted 30 Jan 2024cs.LG cs.AI cs.CLcited by 150

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Despite advances in AI alignment, large language models (LLMs) remain vulnerable to adversarial attacks or jailbreaking, in which adversaries can modify prompts to induce unwanted behavior. While some defenses have been proposed, they have not been adapted to newly proposed attacks and more challenging threat models. To address this, we propose an optimization-based objective for defending LLMs against jailbreaking attacks and an algorithm, Robust Prompt Optimization (RPO) to create robust system-level defenses. Our approach directly incorporates the adversary into the defensive objective and optimizes a lightweight and transferable suffix, enabling RPO to adapt to worst-case adaptive attacks. Our theoretical and experimental results show improved robustness to both jailbreaks seen during optimization and unknown jailbreaks, reducing the attack success rate (ASR) on GPT-4 to 6% and Llama-2 to 0% on JailbreakBench, setting the state-of-the-art. Code can be found at https://github.com/lapisrocks/rpo

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 58 of its 82 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

58 of 82 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  3. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization15 Nov
  4. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  6. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  7. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  8. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  9. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  10. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  11. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
2 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  2. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2013Intriguing properties of neural networkscited by 5,114
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
32 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  3. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  4. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  5. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  7. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  8. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  9. 2017Countering Adversarial Images using Input Transformationscited by 517
  10. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  11. 2022Diffusion Models for Adversarial Purificationcited by 469
  12. 2018Generating Natural Language Adversarial Examplescited by 422
  13. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  14. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  15. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  16. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  17. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 328
  18. 2023Are aligned neural networks adversarially aligned?cited by 305
  19. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  20. 2018Spatially Transformed Adversarial Examplescited by 195
  21. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  22. 2019Adversarial Training and Robustness for Multiple Perturbationscited by 182
  23. 2018Semidefinite relaxations for certifying robustness to adversarial examplescited by 178
  24. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  25. 2020Perceptual Adversarial Robustness: Defense Against Unseen Threat Modelscited by 137
  26. 2019Adversarial Robustness Against the Union of Multiple Perturbation Modelscited by 88
  27. 2020The Radicalization Risks of GPT-3 and Advanced Neural Language Modelscited by 76
  28. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  29. 2022SemAttack: Natural Textual Attacks via Different Semantic Spacescited by 42
  30. 2018Theoretical Analysis of Adversarial Learning: A Minimax Approachcited by 31
  31. 2020Learning to Generate Noise for Multi-Attack Robustnesscited by 17
  32. 2023MultiRobustBench: Benchmarking Robustness Against Multiple Attackscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 53; 2025: 55; 2026: 42.

150 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  3. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  4. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  5. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  6. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  7. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  8. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  9. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  10. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  11. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  12. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
38 more
  1. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  2. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 88
  3. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  4. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 80
  5. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 74
  6. 2024Safeguarding Large Language Models: A Surveycited by 73
  7. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 70
  8. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  9. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  10. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 65
  11. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 63
  12. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  13. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 50
  14. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  15. 2024GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Modelscited by 44
  16. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  17. 2024Boosting Jailbreak Attack with Momentumcited by 38
  18. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 38
  19. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  20. 2025Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreakingcited by 34
  21. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  22. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  23. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  24. 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attackscited by 28
  25. 2025CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMscited by 26
  26. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  27. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
  28. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
  29. 2024Faster-GCG: Efficient Discrete Optimization Jailbreak Attacks against Aligned Large Language Modelscited by 23
  30. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreakingcited by 22
  31. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  32. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  33. 2025ASIDE: Architectural Separation of Instructions and Data in Language Modelscited by 21
  34. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  35. 2024Endless Jailbreaks with Bijection Learningcited by 18
  36. 2024AI Risk Management Should Incorporate Both Safety and Securitycited by 17
  37. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  38. 2024HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Routercited by 16

50 most cited of 150 All 150 (JSON)