mlsanityData to 14 Oct 2026

Fight Back Against Jailbreaking via Prompt Adversarial Tuning

arXiv 2402.06255 ↗First posted 9 Feb 2024cs.LG cs.AI cs.CLcited by 67

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

While Large Language Models (LLMs) have achieved tremendous success in various applications, they are also susceptible to jailbreaking attacks. Several primary defense strategies have been proposed to protect LLMs from producing harmful information, mostly focusing on model fine-tuning or heuristical defense designs. However, how to achieve intrinsic robustness through prompt optimization remains an open problem. In this paper, motivated by adversarial training paradigms for achieving reliable robustness, we propose an approach named Prompt Adversarial Tuning (PAT) that trains a prompt control attached to the user prompt as a guard prefix. To achieve our defense goal whilst maintaining natural performance, we optimize the control prompt with both adversarial and benign prompts. Comprehensive experiments show that our method is effective against both grey-box and black-box attacks, reducing the success rate of advanced attacks to nearly 0%, while maintaining the model's utility on the benign task and incurring only negligible computational overhead, charting a new perspective for future explorations in LLM security. Our code is available at https://github.com/PKU-ML/PAT.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 54 of its 80 references that are in the map. First posted on arXiv 9 Feb 2024.

Builds on

54 of 80 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  2. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Models30 Jan
  3. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  4. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  5. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks16 Oct
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  7. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  8. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  9. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  10. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  11. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  12. 2023Detecting Language Model Attacks with Perplexity27 Aug
3 more
  1. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2023Mistral 7Bcited by 3,164
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  12. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
27 more
  1. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  2. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  4. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  5. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  7. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  8. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  9. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  10. 2023Role-Play with Large Language Modelscited by 294
  11. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  12. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  13. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  14. 2021Unlearnable Examples: Making Personal Data Unexploitablecited by 193
  15. 2023Planning with Large Language Models for Code Generationcited by 182
  16. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 175
  17. 2020Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNetscited by 167
  18. 2021On the Convergence and Robustness of Adversarial Trainingcited by 130
  19. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 116
  20. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 76
  21. 2022When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecturecited by 64
  22. 2023CFA: Class-wise Calibrated Fair Adversarial Trainingcited by 53
  23. 2023Improving Large Language Model Fine-tuning for Solving Math Problemscited by 49
  24. 2022Self-Ensemble Adversarial Training for Improved Robustnesscited by 38
  25. 2023Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attackscited by 34
  26. 2023Generalist: Decoupling Natural and Robust Generalizationcited by 14
  27. 2021Finding Optimal Tangent Points for Reducing Distortions of Hard-label Attackscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 23; 2026: 27.

67 papers.

Sort:

By area:

  1. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 153
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  3. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  4. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 70
  5. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 63
  6. 2024Jailbreak Large Vision-Language Models Through Multi-Modal Linkagecited by 62
  7. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 54
  8. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  9. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 44
  10. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 42
  11. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 34
  12. 2025Defending Against Prompt Injection With a Few DefensiveTokenscited by 33
38 more
  1. 2024On the Duality Between Sharpness-Aware Minimization and Adversarial Trainingcited by 24
  2. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 24
  3. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 21
  4. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 21
  5. 2025CASE-Bench: Context-Aware SafEty Benchmark for Large Language Modelscited by 20
  6. 2024Uncovering Logit Suppression Vulnerabilities in LLM Safety Alignmentcited by 18
  7. 2024Adversarial Representation Engineering: A General Model Editing Framework for Large Language Modelscited by 12
  8. 2025E2AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Modelscited by 12
  9. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMscited by 12
  10. 2025MixAT: Combining Continuous and Discrete Adversarial Training for LLMscited by 11
  11. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 11
  12. 2024LLMs Can Defend Themselves Against Jailbreaking in a Practical Manner: A Vision Papercited by 10
  13. 2025You Can't Eat Your Cake and Have It Too: The Performance Degradation of LLMs with Jailbreak Defensecited by 10
  14. 2025DETAM: Defending LLMs Against Jailbreak Attacks via Targeted Attention Modificationcited by 10
  15. 2026Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMscited by 9
  16. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  17. 2025RAFP: Identifying LLM Lineages via Rare-Region Fingerprintscited by 8
  18. 2025DAVSP: Safety Alignment for Large Vision-Language Models via Deep Aligned Visual Safety Promptcited by 8
  19. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 8
  20. 2026HoneyTrap: Deceiving Large Language Model Attackers to Honeypot Traps with Resilient Multi-Agent Defensecited by 8
  21. 2025Towards the Worst-case Robustness of Large Language Modelscited by 7
  22. 2026Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memorycited by 7
  23. 2025MetaSC: Test-Time Safety Specification Optimization for Language Modelscited by 4
  24. 2025MetaCipher: A Time-Persistent and Universal Multi-Agent Framework for Cipher-Based Jailbreak Attacks for LLMscited by 4
  25. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probingcited by 4
  26. 2025STShield: Single-Token Sentinel for Real-Time Jailbreak Detection in Large Language Modelscited by 3
  27. 2025MIST: Jailbreaking Black-box Large Language Models via Iterative Semantic Tuningcited by 3
  28. 2025Jailbreaking Attacks vs. Content Safety Filters: How Far Are We in the LLM Safety Arms Race?cited by 3
  29. 2026Disclosure By Design: Identity Transparency as a Behavioural Property of Conversational AI Modelscited by 3
  30. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 3
  31. 2024New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlookcited by 2
  32. 2025Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuationcited by 2
  33. 2025Breaking Minds, Breaking Systems: Jailbreaking Large Language Models via Human-like Psychological Manipulationcited by 2
  34. 2026TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Modelscited by 2
  35. 2026Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanismscited by 2
  36. 2026Test-Time Safety Alignmentcited by 2
  37. 2026Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layerscited by 2
  38. 2024On the Adversarial Transferability of Generalized "Skip Connections"cited by 1

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs
  2. 2025Efficient and Stealthy Jailbreak Attacks via Adversarial Prompt Distillation from LLMs to SLMs
  3. 2025Jailbreaking with Universal Multi-Prompts
  4. 2025Adversarial Prompt Evaluation: Systematic Benchmarking of Guardrails Against Prompt Input Attacks on LLMs
  5. 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models
  6. 2024Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation
  7. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compression
  8. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing