mlsanityData to 7 Oct 2026

On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them

arXiv 2006.08403 ↗First posted 15 Jun 2020cs.LG stat.MLcited by 59

Not on the map. Why?

Abstract (arXiv)

We analyze the influence of adversarial training on the loss landscape of machine learning models. To this end, we first provide analytical studies of the properties of adversarial loss functions under different adversarial budgets. We then demonstrate that the adversarial loss landscape is less favorable to optimization, due to increased curvature and more scattered gradients. Our conclusions are validated by numerical analyses, which show that training under large adversarial budgets impede the escape from suboptimal random initialization, cause non-vanishing gradients and make the model find sharper minima. Based on these observations, we show that a periodic adversarial scheduling (PAS) strategy can effectively overcome these challenges, yielding better results than vanilla adversarial training while being much less sensitive to the choice of learning rate.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 19; 2023: 7; 2024: 14; 2025: 6; 2026: 2.

59 papers.

Sort:

  1. 2022Transformers in Medical Imaging: A Surveycited by 245
  2. 2022SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentationcited by 171
  3. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  4. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  5. 2022Sharpness-Aware Training for Freecited by 95
  6. 2021Exploring Memorization in Adversarial Trainingcited by 55
  7. 2021Reliable Adversarial Distillation with Unreliable Teacherscited by 55
  8. 2022MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initializationcited by 31
  9. 2022Stability Analysis and Generalization Bounds of Adversarial Trainingcited by 30
  10. 2021Sharpness-aware Quantization for Deep Neural Networkscited by 25
  11. 2022Exploring the Relationship between Architecture and Adversarially Robust Generalizationcited by 23
  12. 2022Understanding and Combating Robust Overfitting via Input Loss Landscape Analysis and Regularizationcited by 22
38 more
  1. 2023TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalizationcited by 18
  2. 2024Momentum-SAM: Sharpness Aware Minimization without Computational Overheadcited by 13
  3. 2022Why adversarial training can hurt robust accuracycited by 12
  4. 2021On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Trainingcited by 11
  5. 2022How Does Frequency Bias Affect the Robustness of Neural Image Classifiers against Common Corruption and Adversarial Perturbations?cited by 11
  6. 2022Adaptive Smoothness-weighted Adversarial Training for Multiple Perturbations with Its Stability Analysiscited by 9
  7. 2022Defending Black-box Skeleton-based Human Activity Classifierscited by 8
  8. 2024Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Modelscited by 7
  9. 2023RobArch: Designing Robust Architectures against Adversarial Attackscited by 6
  10. 2024PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutorcited by 6
  11. 2024Adversarial Training: A Surveycited by 6
  12. 2021Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regressioncited by 5
  13. 2021Adversarial for Good? How the Adversarial ML Community's Values Impede Socially Beneficial Uses of Attackscited by 5
  14. 2022Robust Binary Models by Pruning Randomly-initialized Networkscited by 5
  15. 2023Focus on Hiders: Exploring Hidden Threats for Enhancing Adversarial Trainingcited by 5
  16. 2022Alleviating Robust Overfitting of Adversarial Training With Consistency Regularizationcited by 4
  17. 2022Why Adversarial Training of ReLU Networks Is Difficult?cited by 4
  18. 2022Adversarial Training with Complementary Labels: On the Benefit of Gradually Informative Attackscited by 4
  19. 2024Data-Dependent Stability Analysis of Adversarial Trainingcited by 4
  20. 2024A direct proof of a unified law of robustness for Bregman divergence lossescited by 4
  21. 2021Bridged Adversarial Trainingcited by 3
  22. 2022Boosting Factorization Machines via Saliency-Guided Mixupcited by 3
  23. 2022Strength-Adaptive Adversarial Trainingcited by 3
  24. 2023A Survey of Historical Learning: Learning Models with Learning Historycited by 3
  25. 2021Smoothness Analysis of Adversarial Trainingcited by 2
  26. 2021Location Leakage in Federated Signal Mapscited by 2
  27. 2023R'enyi Pufferfish Privacy: General Additive Noise Mechanisms and Privacy Amplification by Iterationcited by 2
  28. 2024Uniformly Stable Algorithms for Adversarial Training and Beyondcited by 2
  29. 2024Fisher Information guided Purification against Backdoor Attackscited by 2
  30. 2023Understanding Robust Overfitting from the Feature Generalization Perspectivecited by 1
  31. 2024Improving Adversarial Training using Vulnerability-Aware Perturbation Budgetcited by 1
  32. 2024Effective Gradient Sample Size via Variation Estimation for Accelerating Sharpness aware Minimizationcited by 1
  33. 2024Adversarial Robustness Overestimation and Instability in TRADEScited by 1
  34. 2021On the Noise Stability and Robustness of Adversarially Trained Networks on NVM Crossbarscited by 0
  35. 2022A Novel Plug-and-Play Approach for Adversarially Robust Generalizationcited by 0
  36. 2023Learn from the Past: A Proxy Guided Adversarial Defense Framework with Self Distillation Regularizationcited by 0
  37. 2024Are Classification Robustness and Explanation Robustness Really Strongly Correlated? An Analysis Through Input Loss Landscapecited by 0
  38. 2024DataFreeShield: Defending Adversarial Attacks without Training Datacited by 0

50 most cited of 59 All 59 (JSON)