mlsanityData to 7 Oct 2026

Closing the Distribution Gap in Adversarial Training for LLMs

arXiv 2602.15238 ↗First posted 16 Feb 2026cs.LG cs.AI cs.CRcited by 0

Not on the map. Why?

Abstract (arXiv)

Adversarial training for LLMs is one of the most promising methods to reliably improve robustness against adversaries. However, despite significant progress, models remain vulnerable to simple in-distribution exploits, such as rewriting prompts in the past tense or translating them into other languages. We argue that this persistent fragility stems from a fundamental limitation in current adversarial training algorithms: they minimize adversarial loss on their training set but inadequately cover the data distribution, resulting in vulnerability to seemingly simple attacks. To bridge this gap, we propose Distributional Adversarial Training, DAT. We leverage Diffusion LLMs to approximate the true joint distribution of prompts and responses, enabling generation of diverse, high-likelihood samples that address generalization failures. By combining optimization over the data distribution provided by the diffusion model with continuous adversarial training, DAT achieves substantially higher adversarial robustness than previous methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 44 of its 49 references that are in the map. First posted on arXiv 16 Feb 2026.

Builds on

44 of 49 references.

References · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2025Gemma 3 Technical Reportcited by 1,686
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
32 more
  1. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  2. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  3. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  4. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  5. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  6. 2024A StrongREJECT for Empty Jailbreakscited by 352
  7. 2018Adversarially Robust Generalization Requires More Datacited by 318
  8. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  9. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  10. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
  11. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  12. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  13. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  14. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  15. 2024Attacking Large Language Models with Projected Gradient Descentcited by 91
  16. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  17. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  18. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
  19. 2021On the effectiveness of adversarial training against common corruptionscited by 71
  20. 2024Best-of-N Jailbreakingcited by 71
  21. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  22. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  23. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
  24. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21
  25. 2025LLM-Safety Evaluations Lack Robustnesscited by 16
  26. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objectivecited by 15
  27. 2025MixAT: Combining Continuous and Discrete Adversarial Training for LLMscited by 10
  28. 2025Sampling-aware Adversarial Attacks Against Large Language Modelscited by 9
  29. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectivescited by 8
  30. 2025Diffusion LLMs are Natural Adversaries for any LLMcited by 7
  31. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 5
  32. 2025Fast Proxies for LLM Robustness Evaluationcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Information Theoretic Adversarial Training of Large Language Models
  2. 2025What is Adversarial Training for Diffusion Models?
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models
  4. 2023Raising the Bar for Certified Adversarial Robustness with Diffusion Models
  5. 2025Improved Diffusion-based Generative Model with Better Adversarial Robustness
  6. 2025GCG Attack On A Diffusion LLM
  7. 2025VAULT: Vigilant Adversarial Updates via LLM-Driven Retrieval-Augmented Generation for NLI
  8. 2024Large Language Model Sentinel: LLM Agent for Adversarial Purification