mlsanityData to 7 Oct 2026

Improving Fast Adversarial Training Paradigm: An Example Taxonomy Perspective

arXiv 2408.03944 ↗First posted 22 Jul 2024cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

While adversarial training is an effective defense method against adversarial attacks, it notably increases the training cost. To this end, fast adversarial training (FAT) is presented for efficient training and has become a hot research topic. However, FAT suffers from catastrophic overfitting, which leads to a performance drop compared with multi-step adversarial training. However, the cause of catastrophic overfitting remains unclear and lacks exploration. In this paper, we present an example taxonomy in FAT, which identifies that catastrophic overfitting is caused by the imbalance between the inner and outer optimization in FAT. Furthermore, we investigated the impact of varying degrees of training loss, revealing a correlation between training loss and catastrophic overfitting. Based on these observations, we redesign the loss function in FAT with the proposed dynamic label relaxation to concentrate the loss range and reduce the impact of misclassified examples. Meanwhile, we introduce batch momentum initialization to enhance the diversity to prevent catastrophic overfitting in an efficient manner. Furthermore, we also propose Catastrophic Overfitting aware Loss Adaptation (COLA), which employs a separate training strategy for examples based on their loss degree. Our proposed method, named example taxonomy aware FAT (ETA), establishes an improved paradigm for FAT. Experiment results demonstrate our ETA achieves state-of-the-art performance. Comprehensive experiments on four standard datasets demonstrate the competitiveness of our proposed method.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 48 of its 77 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

48 of 77 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  6. 2021ViViT: A Video Vision Transformercited by 1,610
  7. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  8. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  9. 2022Diffusion Models in Vision: A Surveycited by 1,008
  10. 2016Adversarial Machine Learning at Scalecited by 934
  11. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  12. 2020A Survey on Visual Transformercited by 653
36 more
  1. 2019Adversarial Training for Free!cited by 638
  2. 2020Overfitting in adversarially robust deep learningcited by 504
  3. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 276
  4. 2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 242
  5. 2020Understanding and Improving Fast Adversarial Trainingcited by 213
  6. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  7. 2020Bag of Tricks for Adversarial Trainingcited by 199
  8. 2020Torchattacks: A PyTorch Repository for Adversarial Attackscited by 156
  9. 2020Self-Adaptive Training: beyond Empirical Risk Minimizationcited by 126
  10. 2022LAS-AT: Adversarial Training with Learnable Attack Strategycited by 94
  11. 2020Learnable Boundary Guided Adversarial Trainingcited by 93
  12. 2020Understanding Catastrophic Overfitting in Single-step Adversarial Trainingcited by 75
  13. 2023A Pilot Study of Query-Free Adversarial Attack against Stable Diffusioncited by 75
  14. 2023Content-based Unrestricted Adversarial Attackcited by 73
  15. 2020Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defensescited by 61
  16. 2023CFA: Class-wise Calibrated Fair Adversarial Trainingcited by 53
  17. 2022Make Some Noise: Reliable and Efficient Single-Step Adversarial Trainingcited by 42
  18. 2023An Adaptive Model Ensemble Adversarial Attack for Boosting Adversarial Transferabilitycited by 42
  19. 2021Subspace Adversarial Trainingcited by 41
  20. 2020Single-step Adversarial training with Dropout Schedulingcited by 40
  21. 2021Boosting Fast Adversarial Training with Learnable Adversarial Initializationcited by 40
  22. 2022Prior-Guided Adversarial Initialization for Fast Adversarial Trainingcited by 34
  23. 2023Improving Fast Adversarial Training with Prior-Guided Knowledgecited by 34
  24. 2022Towards Efficient Adversarial Training on Vision Transformerscited by 29
  25. 2023WAT: Improve the Worst-class Robustness in Adversarial Trainingcited by 27
  26. 2021Self-Adversarial Training incorporating Forgery Attention for Image Forgery Localizationcited by 24
  27. 2022Fast Adversarial Training with Adaptive Step Sizecited by 24
  28. 2023On the Over-Memorization During Natural, Robust and Catastrophic Overfittingcited by 19
  29. 2021Mutual Adversarial Training: Learning together is better than going alonecited by 17
  30. 2023The Best Defense is a Good Offense: Adversarial Augmentation against Adversarial Attackscited by 15
  31. 2023Fast Adversarial Training with Smooth Convergencecited by 14
  32. 2021Universal Adversarial Training with Class-Wise Perturbationscited by 13
  33. 2024DAFA: Distance-Aware Fair Adversarial Trainingcited by 11
  34. 2023Fast Propagation is Better: Accelerating Single-Step Adversarial Training via Sampling Subnetworkscited by 8
  35. 2021Push Stricter to Decide Better: A Class-Conditional Feature Adaptive Framework for Improving Adversarial Robustnesscited by 6
  36. 2022Vanilla Feature Distillation for Improving the Accuracy-Robustness Trade-Off in Adversarial Trainingcited by 2

Built on by

Not cited yet.