# Fast Adversarial Training with Smooth Convergence

arXiv: https://arxiv.org/abs/2308.12857 · HTML: https://mlsanity.com/paper/2308.12857/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 51 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [Fast is better than free: Revisiting adversarial training](https://mlsanity.com/paper/2001.03994/) (2020)
- [Square Attack: a query-efficient black-box adversarial attack via random search](https://mlsanity.com/paper/1912.00049/) (2019)
- [Adversarial Training for Free!](https://mlsanity.com/paper/1904.12843/) (2019)
- [Overfitting in adversarially robust deep learning](https://mlsanity.com/paper/2002.11569/) (2020)
- [Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack](https://mlsanity.com/paper/1907.02044/) (2019)
- [Adversarial Examples for Semantic Segmentation and Object Detection](https://mlsanity.com/paper/1703.08603/) (2017)
- [Data Poisoning Attacks Against Federated Learning Systems](https://mlsanity.com/paper/2007.08432/) (2020)
- [Adversarial Attacks and Defenses in Images, Graphs and Text: A Review](https://mlsanity.com/paper/1909.08072/) (2019)
- [Understanding and Improving Fast Adversarial Training](https://mlsanity.com/paper/2007.02617/) (2020)
- [Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attack](https://mlsanity.com/paper/1811.09310/) (2018)
- [Long Text Generation via Adversarial Training with Leaked Information](https://mlsanity.com/paper/1709.08624/) (2017)
- [RayS: A Ray Searching Method for Hard-label Adversarial Attack](https://mlsanity.com/paper/2006.12792/) (2020)
- [Understanding Catastrophic Overfitting in Single-step Adversarial Training](https://mlsanity.com/paper/2010.01799/) (2020)
- [Revisiting and Advancing Fast Adversarial Training Through The Lens of Bi-Level Optimization](https://mlsanity.com/paper/2112.12376/) (2021)
- [Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses](https://mlsanity.com/paper/2011.14969/) (2020)
- [Improving the Generalization of Adversarial Training with Domain Adaptation](https://mlsanity.com/paper/1810.00740/) (2018)
- [Bilateral Adversarial Training: Towards Fast Training of More Robust Models Against Adversarial Attacks](https://mlsanity.com/paper/1811.10716/) (2018)
- [Boosting Fast Adversarial Training with Learnable Adversarial Initialization](https://mlsanity.com/paper/2110.05007/) (2021)
- [Yet Another Intermediate-Level Attack](https://mlsanity.com/paper/2008.08847/) (2020)
- [Prior-Guided Adversarial Initialization for Fast Adversarial Training](https://mlsanity.com/paper/2207.08859/) (2022)
- [Towards Understanding Fast Adversarial Training](https://mlsanity.com/paper/2006.03089/) (2020)

## Built on by (most cited first, 14 of 14)

- [Preventing Catastrophic Overfitting in Fast Adversarial Training: A Bi-level Optimization Perspective](https://mlsanity.com/paper/2407.12443/) (2024)
- [When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Models](https://mlsanity.com/paper/2511.21192/) (2025)
- [Adversarial Training: A Survey](https://mlsanity.com/paper/2410.15042/) (2024)
- [Catastrophic Overfitting: A Potential Blessing in Disguise](https://mlsanity.com/paper/2402.18211/) (2024)
- [Attack Anything: Blind DNNs via Universal Background Adversarial Attack](https://mlsanity.com/paper/2409.00029/) (2024)
- [Improving Fast Adversarial Training via Self-Knowledge Guidance](https://mlsanity.com/paper/2409.17589/) (2024)
- [FastAT Benchmark: A Comprehensive Framework for Fair Evaluation of Fast Adversarial Training Methods](https://mlsanity.com/paper/2604.22853/) (2026)
- [Mitigating Error Amplification in Fast Adversarial Training](https://mlsanity.com/paper/2604.24332/) (2026)
- [Improving Fast Adversarial Training Paradigm: An Example Taxonomy Perspective](https://mlsanity.com/paper/2408.03944/) (2024)
- [Sustainable Self-evolution Adversarial Training](https://mlsanity.com/paper/2412.02270/) (2024)
- [Deep learning models are vulnerable, but adversarial examples are even more vulnerable](https://mlsanity.com/paper/2511.05073/) (2025)
- [Quadratic Upper Bound for Boosting Robustness](https://mlsanity.com/paper/2601.13645/) (2026)
- [Unveiling the Backdoor Mechanism Hidden Behind Catastrophic Overfitting in Fast Adversarial Training](https://mlsanity.com/paper/2604.24350/) (2026)
- [Confusion-Geometry Rebalancing for Long-Tailed Adversarial Training](https://mlsanity.com/paper/2608.09688/) (2026)

## Abstract (arXiv)

Fast adversarial training (FAT) is beneficial for improving the adversarial robustness of neural networks. However, previous FAT work has encountered a significant issue known as catastrophic overfitting when dealing with large perturbation budgets, ie the adversarial robustness of models declines to near zero during training. To address this, we analyze the training process of prior FAT work and observe that catastrophic overfitting is accompanied by the appearance of loss convergence outliers. Therefore, we argue a moderately smooth loss convergence process will be a stable FAT process that solves catastrophic overfitting. To obtain a smooth loss convergence process, we propose a novel oscillatory constraint (dubbed ConvergeSmooth) to limit the loss difference between adjacent epochs. The convergence stride of ConvergeSmooth is introduced to balance convergence and smoothing. Likewise, we design weight centralization without introducing additional hyperparameters other than the loss balance coefficient. Our proposed methods are attack-agnostic and thus can improve the training stability of various FAT techniques. Extensive experiments on popular datasets show that the proposed methods efficiently avoid catastrophic overfitting and outperform all previous FAT methods. Code is available at urlhttps://github.com/FAT-CS/ConvergeSmooth.