# Revisiting Adversarial Training at Scale

arXiv: https://arxiv.org/abs/2401.04727 · HTML: https://mlsanity.com/paper/2401.04727/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 66 references that are in the map.
- First posted on arXiv 9 Jan 2024.

## Builds on (most cited first, 30 of 58)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Fast is better than free: Revisiting adversarial training](https://mlsanity.com/paper/2001.03994/) (2020)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)

## Built on by (most cited first, 30 of 33)

- [Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?](https://mlsanity.com/paper/2407.21792/) (2024)
- [Trading Inference-Time Compute for Adversarial Robustness](https://mlsanity.com/paper/2501.18841/) (2025)
- [TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models](https://mlsanity.com/paper/2411.13136/) (2024)
- [Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness](https://mlsanity.com/paper/2501.09446/) (2025)
- [Transferable Adversarial Attacks on Black-Box Vision-Language Models](https://mlsanity.com/paper/2505.01050/) (2025)
- [Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models](https://mlsanity.com/paper/2502.01576/) (2025)
- [Fast Adversarial Training with Weak-to-Strong Spatial-Temporal Consistency in the Frequency Domain on Videos](https://mlsanity.com/paper/2504.14921/) (2025)
- [AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization](https://mlsanity.com/paper/2504.01735/) (2025)
- [Diffusion-based Adversarial Purification from the Perspective of the Frequency Domain](https://mlsanity.com/paper/2505.01267/) (2025)
- [VEAttack: Downstream-agnostic Vision Encoder Attack against Large Vision Language Models](https://mlsanity.com/paper/2505.17440/) (2025)
- [Adversarial Training: A Survey](https://mlsanity.com/paper/2410.15042/) (2024)
- [Adversarial Prompt Distillation for Vision-Language Models](https://mlsanity.com/paper/2411.15244/) (2024)
- [SoK: Blockchain-Based Decentralized AI (DeAI)](https://mlsanity.com/paper/2411.17461/) (2024)
- [ATP: Adaptive Threshold Pruning for Efficient Data Encoding in Quantum Neural Networks](https://mlsanity.com/paper/2503.21815/) (2025)
- [PBCAT: Patch-based composite adversarial training against physically realizable attacks on object detection](https://mlsanity.com/paper/2506.23581/) (2025)
- [Robustness Feature Adapter for Efficient Adversarial Training](https://mlsanity.com/paper/2508.17680/) (2025)
- [Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP](https://mlsanity.com/paper/2601.19210/) (2026)
- [Boosting Adversarial Transferability with Spatial Adversarial Alignment](https://mlsanity.com/paper/2501.01015/) (2025)
- [Weakly Supervised Contrastive Adversarial Training for Learning Robust Features from Semi-supervised Data](https://mlsanity.com/paper/2503.11032/) (2025)
- [A Few Large Shifts: Layer-Inconsistency Based Minimal Overhead Adversarial Example Detection](https://mlsanity.com/paper/2505.12586/) (2025)
- [A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy](https://mlsanity.com/paper/2508.14079/) (2025)
- [Mitigating Error Amplification in Fast Adversarial Training](https://mlsanity.com/paper/2604.24332/) (2026)
- [DeepDefense: Robust Learning via Layer-Wise Gradient-Feature Alignment](https://mlsanity.com/paper/2511.13749/) (2025)
- [Feature-Space Smoothing: Certified Robustness of Deep Representations](https://mlsanity.com/paper/2601.16200/) (2026)
- [What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models](https://mlsanity.com/paper/2603.12799/) (2026)
- [Unveiling the Backdoor Mechanism Hidden Behind Catastrophic Overfitting in Fast Adversarial Training](https://mlsanity.com/paper/2604.24350/) (2026)
- [Laplace-Bridged Randomized Smoothing for Fast Certified Robustness](https://mlsanity.com/paper/2604.24993/) (2026)
- [TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models](https://mlsanity.com/paper/2605.17577/) (2026)
- [Investigating Adversarial Robustness of Multi-modal Large Language Models](https://mlsanity.com/paper/2606.03713/) (2026)
- [Adaptive Causal Alignment for High-Confidence Adversarial Training](https://mlsanity.com/paper/2606.03925/) (2026)

## Abstract (arXiv)

The machine learning community has witnessed a drastic change in the training pipeline, pivoted by those ''foundation models'' with unprecedented scales. However, the field of adversarial training is lagging behind, predominantly centered around small model sizes like ResNet-50, and tiny and low-resolution datasets like CIFAR-10. To bridge this transformation gap, this paper provides a modern re-examination with adversarial training, investigating its potential benefits when applied at scale. Additionally, we introduce an efficient and effective training strategy to enable adversarial training with giant models and web-scale data at an affordable computing cost. We denote this newly introduced framework as AdvXL. Empirical results demonstrate that AdvXL establishes new state-of-the-art robust accuracy records under AutoAttack on ImageNet-1K. For example, by training on DataComp-1B dataset, our AdvXL empowers a vanilla ViT-g model to substantially surpass the previous records of l∞-, l2-, and l1-robust accuracy by margins of 11.4%, 14.2% and 12.9%, respectively. This achievement posits AdvXL as a pioneering approach, charting a new trajectory for the efficient training of robust visual representations at significantly larger scales. Our code is available at https://github.com/UCSC-VLAA/AdvXL.