mlsanityData to 7 Oct 2026

Investigating Catastrophic Overfitting in Fast Adversarial Training: A Self-fitting Perspective

arXiv 2302.11963 ↗First posted 23 Feb 2023cs.LG cs.AI cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Although fast adversarial training provides an efficient approach for building robust networks, it may suffer from a serious problem known as catastrophic overfitting (CO), where multi-step robust accuracy suddenly collapses to zero. In this paper, we for the first time decouple single-step adversarial examples into data-information and self-information, which reveals an interesting phenomenon called "self-fitting". Self-fitting, i.e., the network learns the self-information embedded in single-step perturbations, naturally leads to the occurrence of CO. When self-fitting occurs, the network experiences an obvious "channel differentiation" phenomenon that some convolution channels accounting for recognizing self-information become dominant, while others for data-information are suppressed. In this way, the network can only recognize images with sufficient self-information and loses generalization ability to other types of data. Based on self-fitting, we provide new insights into the existing methods to mitigate CO and extend CO to multi-step adversarial training. Our findings reveal a self-learning mechanism in adversarial training and open up new perspectives for suppressing different kinds of information to mitigate CO.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 27 of its 33 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

27 of 33 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  8. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  9. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  10. 2019Adversarial Training for Free!cited by 638
  11. 2018Deep learning for time series classification: a reviewcited by 575
  12. 2017Mitigating Adversarial Effects Through Randomizationcited by 382
15 more
  1. 2020Understanding and Improving Fast Adversarial Trainingcited by 213
  2. 2019You Only Propagate Once: Accelerating Adversarial Training via Maximal Principlecited by 181
  3. 2020Understanding Catastrophic Overfitting in Single-step Adversarial Trainingcited by 75
  4. 2020DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensemblescited by 73
  5. 2020Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defensescited by 61
  6. 2019Efficient Adversarial Training with Transferable Adversarial Examplescited by 54
  7. 2017Adversarial and Clean Data Are Not Twinscited by 52
  8. 2020Universal Adversarial Attack on Attention and the Resulting Dataset DAmageNetcited by 45
  9. 2021Subspace Adversarial Trainingcited by 41
  10. 2022Prior-Guided Adversarial Initialization for Fast Adversarial Trainingcited by 34
  11. 2020Efficient Robust Training via Backward Smoothingcited by 29
  12. 2022Adversarial Attack on Attackers: Post-Process to Mitigate Black-Box Score-Based Query Attackscited by 18
  13. 2021Universal Adversarial Training with Class-Wise Perturbationscited by 13
  14. 2020Relevance Attack on Detectorscited by 8
  15. 2022Unifying Gradients to Improve Real-world Robustness for Deep Networkscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 0; 2026: 1.

5 papers.

  1. 2024Efficient local linearity regularization to overcome catastrophic overfittingcited by 7
  2. 2024Adversarial Training: A Surveycited by 6
  3. 2024Catastrophic Overfitting: A Potential Blessing in Disguisecited by 4
  4. 2024Batch-in-Batch: a new adversarial training framework for initial perturbation and sample selectioncited by 0
  5. 2026Unveiling the Backdoor Mechanism Hidden Behind Catastrophic Overfitting in Fast Adversarial Trainingcited by 0