mlsanityData to 7 Oct 2026

Revisiting Outer Optimization in Adversarial Training

arXiv 2209.01199 ↗First posted 2 Sep 2022cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Despite the fundamental distinction between adversarial and natural training (AT and NT), AT methods generally adopt momentum SGD (MSGD) for the outer optimization. This paper aims to analyze this choice by investigating the overlooked role of outer optimization in AT. Our exploratory evaluations reveal that AT induces higher gradient norm and variance compared to NT. This phenomenon hinders the outer optimization in AT since the convergence rate of MSGD is highly dependent on the variance of the gradients. To this end, we propose an optimization method called ENGM which regularizes the contribution of each input example to the average mini-batch gradients. We prove that the convergence rate of ENGM is independent of the variance of the gradients, and thus, it is suitable for AT. We introduce a trick to reduce the computational cost of ENGM using empirical observations on the correlation between the norm of gradients w.r.t. the network parameters and input examples. Our extensive evaluations and ablation studies on CIFAR-10, CIFAR-100, and TinyImageNet demonstrate that ENGM and its variants consistently improve the performance of a wide range of AT methods. Furthermore, ENGM alleviates major shortcomings of AT including robust overfitting and high sensitivity to hyperparameter settings.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 2 Sep 2022.

Builds on

36 of 50 references.

References · 36

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2016Wide Residual Networkscited by 3,057
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  6. 2016Adversarial examples in the physical worldcited by 1,762
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  9. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  10. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  11. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  12. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
24 more
  1. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  2. 2018Robustness May Be at Odds with Accuracycited by 873
  3. 2015The Limitations of Deep Learning in Adversarial Settingscited by 784
  4. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  5. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  6. 2017Exploring Generalization in Deep Learningcited by 595
  7. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  8. 2020Overfitting in adversarially robust deep learningcited by 504
  9. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 275
  10. 2020Attacks Which Do Not Kill Training Make Adversarial Learning Strongercited by 242
  11. 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
  12. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  13. 2020Bag of Tricks for Adversarial Trainingcited by 199
  14. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 178
  15. 2018Robustness via curvature regularization, and vice versacited by 174
  16. 2019Understanding Adversarial Attacks on Deep Learning Based Medical Image Analysis Systemscited by 151
  17. 2020Self-Adaptive Training: beyond Empirical Risk Minimizationcited by 126
  18. 2019Adversarial Examples Are a Natural Consequence of Test Error in Noisecited by 114
  19. 2020Learnable Boundary Guided Adversarial Trainingcited by 93
  20. 2018A Unified Analysis of Stochastic Momentum Methods for Deep Learningcited by 54
  21. 2019Efficient Adversarial Training with Transferable Adversarial Examplescited by 54
  22. 2020Efficient Robust Training via Backward Smoothingcited by 29
  23. 2019SmoothFool: An Efficient Framework for Computing Smooth Adversarial Perturbationscited by 12
  24. 2020Stochastic Normalized Gradient Descent with Momentum for Large-Batch Trainingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 1; 2025: 0; 2026: 1.

4 papers.

  1. 2023Defenses in Adversarial Machine Learning: A Surveycited by 14
  2. 2024Edge Detectors Can Make Deep Convolutional Neural Networks More Robustcited by 2
  3. 2026When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Studycited by 1
  4. 2023Improving the Robustness of Deep Convolutional Neural Networks Through Feature Learningcited by 0