# Improving Fast Adversarial Training Paradigm: An Example Taxonomy Perspective

arXiv: https://arxiv.org/abs/2408.03944 · HTML: https://mlsanity.com/paper/2408.03944/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 48 of its 77 references that are in the map.
- First posted on arXiv 22 Jul 2024.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Fast is better than free: Revisiting adversarial training](https://mlsanity.com/paper/2001.03994/) (2020)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [Adversarial Training for Free!](https://mlsanity.com/paper/1904.12843/) (2019)
- [Overfitting in adversarially robust deep learning](https://mlsanity.com/paper/2002.11569/) (2020)
- [Recent Advances in Adversarial Training for Adversarial Robustness](https://mlsanity.com/paper/2102.01356/) (2021)
- [Attacks Which Do Not Kill Training Make Adversarial Learning Stronger](https://mlsanity.com/paper/2002.11242/) (2020)
- [Understanding and Improving Fast Adversarial Training](https://mlsanity.com/paper/2007.02617/) (2020)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Bag of Tricks for Adversarial Training](https://mlsanity.com/paper/2010.00467/) (2020)
- [Torchattacks: A PyTorch Repository for Adversarial Attacks](https://mlsanity.com/paper/2010.01950/) (2020)
- [Self-Adaptive Training: beyond Empirical Risk Minimization](https://mlsanity.com/paper/2002.10319/) (2020)
- [LAS-AT: Adversarial Training with Learnable Attack Strategy](https://mlsanity.com/paper/2203.06616/) (2022)
- [Learnable Boundary Guided Adversarial Training](https://mlsanity.com/paper/2011.11164/) (2020)
- [Understanding Catastrophic Overfitting in Single-step Adversarial Training](https://mlsanity.com/paper/2010.01799/) (2020)
- [A Pilot Study of Query-Free Adversarial Attack against Stable Diffusion](https://mlsanity.com/paper/2303.16378/) (2023)
- [Content-based Unrestricted Adversarial Attack](https://mlsanity.com/paper/2305.10665/) (2023)
- [Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses](https://mlsanity.com/paper/2011.14969/) (2020)
- [CFA: Class-wise Calibrated Fair Adversarial Training](https://mlsanity.com/paper/2303.14460/) (2023)
- [Make Some Noise: Reliable and Efficient Single-Step Adversarial Training](https://mlsanity.com/paper/2202.01181/) (2022)
- [An Adaptive Model Ensemble Adversarial Attack for Boosting Adversarial Transferability](https://mlsanity.com/paper/2308.02897/) (2023)

## Abstract (arXiv)

While adversarial training is an effective defense method against adversarial attacks, it notably increases the training cost. To this end, fast adversarial training (FAT) is presented for efficient training and has become a hot research topic. However, FAT suffers from catastrophic overfitting, which leads to a performance drop compared with multi-step adversarial training. However, the cause of catastrophic overfitting remains unclear and lacks exploration. In this paper, we present an example taxonomy in FAT, which identifies that catastrophic overfitting is caused by the imbalance between the inner and outer optimization in FAT. Furthermore, we investigated the impact of varying degrees of training loss, revealing a correlation between training loss and catastrophic overfitting. Based on these observations, we redesign the loss function in FAT with the proposed dynamic label relaxation to concentrate the loss range and reduce the impact of misclassified examples. Meanwhile, we introduce batch momentum initialization to enhance the diversity to prevent catastrophic overfitting in an efficient manner. Furthermore, we also propose Catastrophic Overfitting aware Loss Adaptation (COLA), which employs a separate training strategy for examples based on their loss degree. Our proposed method, named example taxonomy aware FAT (ETA), establishes an improved paradigm for FAT. Experiment results demonstrate our ETA achieves state-of-the-art performance. Comprehensive experiments on four standard datasets demonstrate the competitiveness of our proposed method.