mlsanityData to 14 Oct 2026

Understanding Catastrophic Overfitting in Adversarial Training

arXiv 2105.02942 ↗First posted 6 May 2021cs.LG cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Recently, FGSM adversarial training is found to be able to train a robust model which is comparable to the one trained by PGD but an order of magnitude faster. However, there is a failure mode called catastrophic overfitting (CO) that the classifier loses its robustness suddenly during the training and hardly recovers by itself. In this paper, we find CO is not only limited to FGSM, but also happens in mboxDF∞-1 adversarial training. Then, we analyze the geometric properties for both FGSM and mboxDF∞-1 and find they have totally different decision boundaries after CO. For FGSM, a new decision boundary is generated along the direction of perturbation and makes the small perturbation more effective than the large one. While for mboxDF∞-1, there is no new decision boundary generated along the direction of perturbation, instead the perturbation generated by mboxDF∞-1 becomes smaller after CO and thus loses its effectiveness. We also experimentally analyze three hypotheses on potential factors causing CO. And then based on the empirical analysis, we modify the RS-FGSM by not projecting perturbation back to the l_∞ ball. By this small modification, we could achieve 47.56 ± 0.37% PGD-50-10 accuracy on CIFAR10 with ε=8/255 in contrast to 43.57 ± 0.30% by RS-FGSM and also further extend the working range of ε from 8/255 to 11/255 on CIFAR10 without CO occurring.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 32 of its 34 references that are in the map. First posted on arXiv 6 May 2021.

Builds on

32 of 34 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2016Adversarial examples in the physical worldcited by 1,769
  6. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  7. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  8. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  9. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,318
  10. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  11. 2016Adversarial Machine Learning at Scalecited by 936
  12. 2017Ensemble Adversarial Training: Attacks and Defensescited by 840
20 more
  1. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
  2. 2017Evasion Attacks against Machine Learning at Test Timecited by 689
  3. 2019Adversarial Training for Free!cited by 643
  4. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 567
  5. 2017Countering Adversarial Images using Input Transformationscited by 520
  6. 2020Overfitting in adversarially robust deep learningcited by 506
  7. 2018Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Modelscited by 362
  8. 2018On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Modelscited by 328
  9. 2018Feature Denoising for Improving Adversarial Robustnesscited by 314
  10. 2018Certified Defenses against Adversarial Examplescited by 306
  11. 2017PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examplescited by 244
  12. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  13. 2020Understanding and Improving Fast Adversarial Trainingcited by 214
  14. 2019You Only Propagate Once: Accelerating Adversarial Training via Maximal Principlecited by 183
  15. 2018Robustness via curvature regularization, and vice versacited by 175
  16. 2019Adversarial Robustness through Local Linearizationcited by 146
  17. 2020Understanding Catastrophic Overfitting in Single-step Adversarial Trainingcited by 76
  18. 2020Single-step Adversarial training with Dropout Schedulingcited by 41
  19. 2020Towards Understanding Fast Adversarial Trainingcited by 33
  20. 2018Logit Pairing Methods Can Fool Gradient-Based Attackscited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 2; 2024: 2; 2025: 1; 2026: 2.

12 papers.

  1. 2022Robustness and Accuracy Could Be Reconcilable by (Proper) Definitioncited by 111
  2. 2022Make Some Noise: Reliable and Efficient Single-Step Adversarial Trainingcited by 43
  3. 2021Subspace Adversarial Trainingcited by 41
  4. 2024Adversarial Training: A Surveycited by 6
  5. 2024Sparse-PGD: A Unified Framework for Sparse Adversarial Perturbations Generationcited by 4
  6. 2022Catastrophic overfitting can be induced with discriminative non-robust featurescited by 2
  7. 2022Adversarial amplitude swap towards robust image classifierscited by 1
  8. 2023Intrinsic Biologically Plausible Adversarial Robustnesscited by 1
  9. 2023Exploiting Frequency Spectrum of Adversarial Images for General Robustnesscited by 0
  10. 2025Fast Adversarial Training against Sparse Attacks Requires Loss Smoothingcited by 0
  11. 2026Quadratic Upper Bound for Boosting Robustnesscited by 0
  12. 2026SORA: Free Second-Order Attacks in Fast Adversarial Trainingcited by 0