mlsanityData to 14 Oct 2026

Why Adversarial Training of ReLU Networks Is Difficult?

arXiv 2205.15130 ↗First posted 30 May 2022cs.LG cs.AI cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

This paper mathematically derives an analytic solution of the adversarial perturbation on a ReLU network, and theoretically explains the difficulty of adversarial training. Specifically, we formulate the dynamics of the adversarial perturbation generated by the multi-step attack, which shows that the adversarial perturbation tends to strengthen eigenvectors corresponding to a few top-ranked eigenvalues of the Hessian matrix of the loss w.r.t. the input. We also prove that adversarial training tends to strengthen the influence of unconfident input samples with large gradient norms in an exponential manner. Besides, we find that adversarial training strengthens the influence of the Hessian matrix of the loss w.r.t. network parameters, which makes the adversarial training more likely to oscillate along directions of a few samples, and boosts the difficulty of adversarial training. Crucially, our proofs provide a unified explanation for previous findings in understanding adversarial training.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 30 May 2022.

Builds on

24 of 32 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  6. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  7. 2016Adversarial Machine Learning at Scalecited by 936
  8. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 920
  9. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  10. 2018Robustness May Be at Odds with Accuracycited by 880
  11. 2017Ensemble Adversarial Training: Attacks and Defensescited by 840
  12. 2020Fast is better than free: Revisiting adversarial trainingcited by 727
12 more
  1. 2020Overfitting in adversarially robust deep learningcited by 506
  2. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  3. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 337
  4. 2018Adversarially Robust Generalization Requires More Datacited by 318
  5. 2019Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Trainingcited by 95
  6. 2019Adversarially Robust Generalization Just Requires More Unlabeled Datacited by 76
  7. 2020On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Themcited by 59
  8. 2021On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Trainingcited by 11
  9. 2018Interpreting Adversarial Robustness: A View from Decision Surface in Input Spacecited by 10
  10. 2021Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regressioncited by 5
  11. 2021Smoothness Analysis of Adversarial Trainingcited by 2
  12. 2022Towards Theoretical Analysis of Transformation Complexity of ReLU DNNscited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 3.

4 papers.

  1. 2025RL-Selector: Reinforcement Learning-Guided Data Selection via Redundancy Assessmentcited by 3
  2. 2025IPF-RDA: An Information-Preserving Framework for Robust Data Augmentationcited by 1
  3. 2022Lower Difficulty and Better Robustness: A Bregman Divergence Perspective for Adversarial Trainingcited by 0
  4. 2025Sharpness-Aware Geometric Defense for Robust Out-Of-Distribution Detectioncited by 0