mlsanityData to 7 Oct 2026

Convergence of Adversarial Training in Overparametrized Neural Networks

arXiv 1906.07916 ↗First posted 19 Jun 2019cs.LG stat.MLcited by 58

Not on the map. Why?

Abstract (arXiv)

Neural networks are vulnerable to adversarial examples, i.e. inputs that are imperceptibly perturbed from natural data and yet incorrectly classified by the network. Adversarial training, a heuristic form of robust optimization that alternates between minimization and maximization steps, has proven to be among the most successful methods to train networks to be robust against a pre-defined family of perturbations. This paper provides a partial answer to the success of adversarial training, by showing that it converges to a network where the surrogate loss with respect to the the attack algorithm is within ε of the optimal robust loss. Then we show that the optimal robust loss is also close to zero, hence adversarial training finds a robust classifier. The analysis technique leverages recent work on the analysis of neural networks via Neural Tangent Kernel (NTK), combined with motivation from online-learning when the maximization is solved by a heuristic, and the expressiveness of the NTK kernel in the ℓ_∞-norm. In addition, we also prove that robust interpolation requires more model capacity, supporting the evidence that adversarial training requires wider networks.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 19; 2023: 11; 2024: 9; 2025: 3; 2026: 3.

58 papers.

Sort:

By area:

  1. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  2. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  3. 2021Advances in adversarial attacks and defenses in computer vision: A surveycited by 127
  4. 2021Exploring Memorization in Adversarial Trainingcited by 55
  5. 2021A Review of Adversarial Attack and Defense for Classification Methodscited by 29
  6. 2022Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Powercited by 23
  7. 2021Taxonomy of Machine Learning Safety: A Survey and Primercited by 22
  8. 2021On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Gamecited by 20
  9. 2022Robustness in deep learning: The good (width), the bad (depth), and the ugly (initialization)cited by 18
  10. 2022Federated Adversarial Learning: A Framework with Convergence Analysiscited by 17
  11. 2023Defenses in Adversarial Machine Learning: A Surveycited by 14
  12. 2022Towards Adversarially Robust Deep Image Denoisingcited by 13
38 more
  1. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  2. 2024Nonparametric Teaching of Implicit Neural Representationscited by 13
  3. 2022Understanding Robust Learning through the Lens of Representation Similaritiescited by 12
  4. 2021Benign Overfitting in Adversarially Robust Linear Classificationcited by 11
  5. 2021Provable Robustness of Adversarial Training for Learning Halfspaces with Noisecited by 10
  6. 2022On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimescited by 10
  7. 2022A Law of Robustness beyond Isoperimetrycited by 9
  8. 2022Distributed Adversarial Training to Robustify Deep Neural Networks at Scalecited by 9
  9. 2022Adaptive Smoothness-weighted Adversarial Training for Multiple Perturbations with Its Stability Analysiscited by 9
  10. 2023Theoretical Analysis of Robust Overfitting for Wide DNNs: An NTK Approachcited by 8
  11. 2023Understanding Noise-Augmented Training for Randomized Smoothingcited by 7
  12. 2024The Surprising Harmfulness of Benign Overfitting for Adversarial Robustnesscited by 7
  13. 2021Improving the Robustness of Adversarial Attacks Using an Affine-Invariant Gradient Estimatorcited by 6
  14. 2021Provable Regret Bounds for Deep Online Learning and Controlcited by 6
  15. 2022Adversarial Robustness is at Odds with Lazy Trainingcited by 6
  16. 2024Robust NAS under adversarial training: benchmark, theory, and beyondcited by 6
  17. 2021Robust Generalization of Quadratic Neural Networks via Function Identificationcited by 5
  18. 2022On the Convergence of Certified Robust Training with Interval Bound Propagationcited by 5
  19. 2022Randomly Initialized One-Layer Neural Networks Make Data Linearly Separablecited by 5
  20. 2023On the Clean Generalization and Robust Overfitting in Adversarial Training from Two Theoretical Views: Representation Complexity and Training Dynamicscited by 5
  21. 2022BiFeat: Supercharge GNN Training via Graph Feature Quantizationcited by 4
  22. 2023Exploring Adversarial Attacks on Neural Networks: An Explainable Approachcited by 4
  23. 2024A direct proof of a unified law of robustness for Bregman divergence lossescited by 4
  24. 2021Adversarial Training for Gradient Descent: Analysis Through its Continuous-time Approximationcited by 3
  25. 2022Origins of Low-dimensional Adversarial Perturbationscited by 3
  26. 2023VIPeR: Provably Efficient Algorithm for Offline RL with Neural Function Approximationcited by 3
  27. 2023Generating Less Certain Adversarial Examples Improves Robust Generalizationcited by 3
  28. 2024Attack Anything: Blind DNNs via Universal Background Adversarial Attackcited by 3
  29. 2025Geometry of Neural Reinforcement Learning in Continuous State and Action Spacescited by 3
  30. 2021A Law of Robustness for Weight-bounded Neural Networkscited by 2
  31. 2022Robust Sensible Adversarial Learning of Deep Neural Networks for Image Classificationcited by 2
  32. 2023NoiseCAM: Explainable AI for the Boundary Between Noise and Adversarial Attackscited by 2
  33. 2023Understanding Overfitting in Adversarial Training via Kernel Regressioncited by 2
  34. 2024Spectral regularization for adversarially-robust representation learningcited by 2
  35. 2024Over-parameterization and Adversarial Robustness in Neural Networks: An Overview and Empirical Analysiscited by 2
  36. 2022Finding Dynamics Preserving Adversarial Winning Ticketscited by 1
  37. 2022Semi-Implicit Hybrid Gradient Methods with Application to Adversarial Robustnesscited by 1
  38. 2022Characterizing Internal Evasion Attacks in Federated Learningcited by 1

50 most cited of 58 All 58 (JSON)