mlsanityData to 14 Oct 2026

Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence

arXiv 2502.04204 ↗First posted 6 Feb 2025cs.LG cs.CR stat.MLcited by 14

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Jailbreak attacks against large language models (LLMs) aim to induce harmful behaviors in LLMs through carefully crafted adversarial prompts. To mitigate attacks, one way is to perform adversarial training (AT)-based alignment, i.e., training LLMs on some of the most adversarial prompts to help them learn how to behave safely under attacks. During AT, the length of adversarial prompts plays a critical role in the robustness of aligned LLMs. While long-length adversarial prompts during AT might lead to strong LLM robustness, their synthesis however is very resource-consuming, which may limit the application of LLM AT. This paper focuses on adversarial suffix jailbreak attacks and unveils that to defend against a jailbreak attack with an adversarial suffix of length Theta(M), it is enough to align LLMs on prompts with adversarial suffixes of length Theta(√M). Theoretically, we analyze the adversarial in-context learning of linear transformers on linear regression tasks and prove a robust generalization bound for trained transformers. The bound depends on the term Theta(√Mtest/Mtrain), where Mtrain and Mtest are the numbers of adversarially perturbed in-context samples during training and testing. Empirically, we conduct AT on popular open-source LLMs and evaluate their robustness against jailbreak attacks of different adversarial suffix lengths. Results confirm a positive correlation between the attack success rate and the ratio of the square root of the adversarial suffix length during jailbreaking to the length during AT. Our findings show that it is practical to defend against "long-length" jailbreak attacks via efficient "short-length" AT. The code is available at https://github.com/fshp971/adv-icl.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 59 of its 75 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

59 of 75 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs3 Oct
  2. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs22 Jul
  3. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs13 Jun
  4. 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters30 May
  5. 2024Efficient Adversarial Training in LLMs with Continuous Attacks24 May
  6. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  7. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
  8. 2024Fast Adversarial Attacks on Language Models In One GPU Minute23 Feb
  9. 2024Query-Based Adversarial Prompt Generation19 Feb
  10. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space14 Feb
  11. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  12. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
5 more
  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  2. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  5. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  10. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  11. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  12. 2013Intriguing properties of neural networkscited by 5,155
30 more
  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2023Mistral 7Bcited by 3,164
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  7. 2022Transformers learn in-context by gradient descentcited by 732
  8. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  9. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  10. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  11. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  12. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  13. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  14. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  15. 2023In-Context Convergence of Transformerscited by 115
  16. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
  17. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  18. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 88
  19. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  20. 2020Precise Tradeoffs in Adversarial Training for Linear Regressioncited by 77
  21. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 64
  22. 2023Adversarial Demonstration Attacks on Large Language Modelscited by 63
  23. 2024Asymptotic theory of in-context learning by linear attentioncited by 55
  24. 2024Why Larger Language Models Do In-context Learning Differently?cited by 45
  25. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  26. 2023Regularization properties of adversarially-trained linear regressioncited by 18
  27. 2024Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Contextcited by 15
  28. 2024How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regressioncited by 14
  29. 2024Benign Overfitting in Single-Head Attentioncited by 12
  30. 2023Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approachcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

By area:

  1. 2025Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducementscited by 32
  2. 2025Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Unlearning Completenesscited by 11
  3. 2025Adversarially Pretrained Transformers May Be Universally Robust In-Context Learnerscited by 3
  4. 2025Stable Vision Concept Transformers for Medical Diagnosiscited by 3
  5. 2025Attributing Data for Sharpness-Aware Minimizationcited by 2
  6. 2026Understanding the Dynamics of Demonstration Conflict in In-Context Learningcited by 2
  7. 2026Word Recovery in Large Language Models Enables Character-Level Tokenization Robustnesscited by 2
  8. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  9. 2026Differentially Private Non-convex Distributionally Robust Optimizationcited by 1
  10. 2026Understanding and Improving Continuous Adversarial Training for LLMs via In-context Learning Theorycited by 1
  11. 2026Benign Overfitting in Adversarial Training for Vision Transformerscited by 1
  12. 2026Towards Understanding the Robustness of Sparse Autoencoderscited by 0
2 more
  1. 2026Information Theoretic Adversarial Training of Large Language Modelscited by 0
  2. 2026Algorithmic Recourse of In-Context Learning for Tabular Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs
  2. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  3. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM
  4. 2025Adversarial Reasoning at Jailbreaking Time
  5. 2025Scaling Patterns in Adversarial Alignment: Evidence from Multi-LLM Jailbreak Experiments
  6. 2025Efficient and Stealthy Jailbreak Attacks via Adversarial Prompt Distillation from LLMs to SLMs
  7. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey
  8. 2024Defending against Jailbreak through Early Exit Generation of Large Language Models