mlsanityData to 7 Oct 2026

Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs

arXiv 2406.06622 ↗First posted 7 Jun 2024cs.CL cs.AI cs.CRcited by 23

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Although safely enhanced Large Language Models (LLMs) have achieved remarkable success in tackling various complex tasks in a zero-shot manner, they remain susceptible to jailbreak attacks, particularly the unknown jailbreak attack. To enhance LLMs' generalized defense capabilities, we propose a two-stage adversarial tuning framework, which generates adversarial prompts to explore worst-case scenarios by optimizing datasets containing pairs of adversarial prompts and their safe responses. In the first stage, we introduce the hierarchical meta-universal adversarial prompt learning to efficiently and effectively generate token-level adversarial prompts. In the second stage, we propose the automatic adversarial prompt learning to iteratively refine semantic-level adversarial prompts, further enhancing LLM's defense capabilities. We conducted comprehensive experiments on three widely used jailbreak datasets, comparing our framework with six defense baselines under five representative attack scenarios. The results underscore the superiority of our proposed methods. Furthermore, our adversarial tuning framework exhibits empirical generalizability across various attack strategies and target LLMs, highlighting its potential as a transferable defense mechanism.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 47 of its 54 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

47 of 54 references.

Closest ancestors · 31

Same direction, newest first

  1. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses3 Jun
  2. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Models31 May
  3. 2024Efficient Adversarial Training in LLMs with Continuous Attacks24 May
  4. 2024Protecting Your LLMs with Information Bottleneck22 Apr
  5. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  6. 2024JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models12 Apr
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  8. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack2 Apr
  9. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  10. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks2 Mar
  11. 2024Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes1 Mar
  12. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction28 Feb
19 more
  1. 2024Defending LLMs against Jailbreaking Attacks via Backtranslation26 Feb
  2. 2024Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing25 Feb
  3. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails24 Feb
  4. 2024Is the System Message Really Important to Jailbreaks in Large Language Models?20 Feb
  5. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  6. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMs8 Feb
  7. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  8. 2024Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning19 Jan
  9. 2023Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak7 Dec
  10. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  12. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  13. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  14. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  15. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  16. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  17. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  18. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  19. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 16

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  4. 2016Universal adversarial perturbationscited by 774
  5. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  6. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  7. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  8. 2023Large Language Model Unlearningcited by 288
  9. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  10. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  11. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  12. 2024Model Extrapolation Expedites Alignmentcited by 37
4 more
  1. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  2. 2024C-RAG: Certified Generation Risks for Retrieval-Augmented Language Modelscited by 27
  3. 2024BELLS: A Framework Towards Future Proof Benchmarks for the Evaluation of LLM Safeguardscited by 13
  4. 2024Cross-Task Defense: Instruction-Tuning LLMs for Content Safetycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 16; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  2. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  3. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  4. 2025EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safetycited by 21
  5. 2025Bag of Tricks for Inference-time Computation of LLM Reasoningcited by 15
  6. 2025To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Promptcited by 14
  7. 2025DETAM: Defending LLMs Against Jailbreak Attacks via Targeted Attention Modificationcited by 10
  8. 2025MixAT: Combining Continuous and Discrete Adversarial Training for LLMscited by 10
  9. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  10. 2025Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attackscited by 7
  11. 2025A Survey on Proactive Defense Strategies Against Misinformation in Large Language Modelscited by 5
  12. 2025Large Language Model Powered Intelligent Urban Agents: Concepts, Capabilities, and Applicationscited by 4
11 more
  1. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  2. 2025STShield: Single-Token Sentinel for Real-Time Jailbreak Detection in Large Language Modelscited by 3
  3. 2026AISA: Awakening Intrinsic Safety Awareness in Large Language Models against Jailbreak Attackscited by 3
  4. 2025Attention-Aware GNN-based Input Defense against Multi-Turn LLM Jailbreakcited by 2
  5. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representationscited by 2
  6. 2025Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attackscited by 1
  7. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  8. 2025RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositoriescited by 0
  9. 2025SoK: a Comprehensive Causality Analysis Framework for Large Language Model Securitycited by 0
  10. 2026LLM-VA: Resolving the Jailbreak-Overrefusal Trade-off via Vector Alignmentcited by 0
  11. 2026Closing the Distribution Gap in Adversarial Training for LLMscited by 0