# On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them

arXiv: https://arxiv.org/abs/2006.08403 · HTML: https://mlsanity.com/paper/2006.08403/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2020.

## Built on by (most cited first, 30 of 59)

- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation](https://mlsanity.com/paper/2202.03104/) (2022)
- [Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation](https://mlsanity.com/paper/2211.11004/) (2022)
- [Efficient Sharpness-aware Minimization for Improved Training of Neural Networks](https://mlsanity.com/paper/2110.03141/) (2021)
- [Sharpness-Aware Training for Free](https://mlsanity.com/paper/2205.14083/) (2022)
- [Reliable Adversarial Distillation with Unreliable Teachers](https://mlsanity.com/paper/2106.04928/) (2021)
- [Exploring Memorization in Adversarial Training](https://mlsanity.com/paper/2106.01606/) (2021)
- [MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initialization](https://mlsanity.com/paper/2210.00102/) (2022)
- [Stability Analysis and Generalization Bounds of Adversarial Training](https://mlsanity.com/paper/2210.00960/) (2022)
- [Sharpness-aware Quantization for Deep Neural Networks](https://mlsanity.com/paper/2111.12273/) (2021)
- [Exploring the Relationship between Architecture and Adversarially Robust Generalization](https://mlsanity.com/paper/2209.14105/) (2022)
- [Understanding and Combating Robust Overfitting via Input Loss Landscape Analysis and Regularization](https://mlsanity.com/paper/2212.04985/) (2022)
- [TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization](https://mlsanity.com/paper/2303.11135/) (2023)
- [Why adversarial training can hurt robust accuracy](https://mlsanity.com/paper/2203.02006/) (2022)
- [Momentum-SAM: Sharpness Aware Minimization without Computational Overhead](https://mlsanity.com/paper/2401.12033/) (2024)
- [On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Training](https://mlsanity.com/paper/2112.07324/) (2021)
- [How Does Frequency Bias Affect the Robustness of Neural Image Classifiers against Common Corruption and Adversarial Perturbations?](https://mlsanity.com/paper/2205.04533/) (2022)
- [Defending Black-box Skeleton-based Human Activity Classifiers](https://mlsanity.com/paper/2203.04713/) (2022)
- [Adaptive Smoothness-weighted Adversarial Training for Multiple Perturbations with Its Stability Analysis](https://mlsanity.com/paper/2210.00557/) (2022)
- [Focus on Hiders: Exploring Hidden Threats for Enhancing Adversarial Training](https://mlsanity.com/paper/2312.07067/) (2023)
- [PeerAiD: Improving Adversarial Distillation from a Specialized Peer Tutor](https://mlsanity.com/paper/2403.06668/) (2024)
- [Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models](https://mlsanity.com/paper/2406.13137/) (2024)
- [RobArch: Designing Robust Architectures against Adversarial Attacks](https://mlsanity.com/paper/2301.03110/) (2023)
- [Adversarial Training: A Survey](https://mlsanity.com/paper/2410.15042/) (2024)
- [Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression](https://mlsanity.com/paper/2102.02950/) (2021)
- [Adversarial for Good? How the Adversarial ML Community's Values Impede Socially Beneficial Uses of Attacks](https://mlsanity.com/paper/2107.10302/) (2021)
- [Robust Binary Models by Pruning Randomly-initialized Networks](https://mlsanity.com/paper/2202.01341/) (2022)
- [Alleviating Robust Overfitting of Adversarial Training With Consistency Regularization](https://mlsanity.com/paper/2205.11744/) (2022)
- [Why Adversarial Training of ReLU Networks Is Difficult?](https://mlsanity.com/paper/2205.15130/) (2022)
- [Adversarial Training with Complementary Labels: On the Benefit of Gradually Informative Attacks](https://mlsanity.com/paper/2211.00269/) (2022)

## Abstract (arXiv)

We analyze the influence of adversarial training on the loss landscape of machine learning models. To this end, we first provide analytical studies of the properties of adversarial loss functions under different adversarial budgets. We then demonstrate that the adversarial loss landscape is less favorable to optimization, due to increased curvature and more scattered gradients. Our conclusions are validated by numerical analyses, which show that training under large adversarial budgets impede the escape from suboptimal random initialization, cause non-vanishing gradients and make the model find sharper minima. Based on these observations, we show that a periodic adversarial scheduling (PAS) strategy can effectively overcome these challenges, yielding better results than vanilla adversarial training while being much less sensitive to the choice of learning rate.