mlsanityData to 14 Oct 2026

Adversarial Training Makes Weight Loss Landscape Sharper in Logistic Regression

arXiv 2102.02950 ↗First posted 5 Feb 2021stat.ML cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Adversarial training is actively studied for learning robust models against adversarial examples. A recent study finds that adversarially trained models degenerate generalization performance on adversarial examples when their weight loss landscape, which is loss changes with respect to weights, is sharp. Unfortunately, it has been experimentally shown that adversarial training sharpens the weight loss landscape, but this phenomenon has not been theoretically clarified. Therefore, we theoretically analyze this phenomenon in this paper. As a first step, this paper proves that adversarial training with the L2 norm constraints sharpens the weight loss landscape in the linear logistic regression model. Our analysis reveals that the sharpness of the weight loss landscape is caused by the noise aligned in the direction of increasing the loss, which is used in adversarial training. We theoretically and experimentally confirm that the weight loss landscape becomes sharper as the magnitude of the noise of adversarial training increases in the linear logistic regression model. Moreover, we experimentally confirm the same phenomena in ResNet18 with softmax as a more general case.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 26 of its 41 references that are in the map. First posted on arXiv 5 Feb 2021.

Builds on

26 of 41 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2013Intriguing properties of neural networkscited by 5,155
  6. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  7. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  8. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  10. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,340
  11. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  12. 2016Adversarial Machine Learning at Scalecited by 936
14 more
  1. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
  2. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  3. 2020Overfitting in adversarially robust deep learningcited by 506
  4. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  5. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  6. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 330
  7. 2017On Detecting Adversarial Perturbationscited by 232
  8. 2020Randomized Smoothing of All Shapes and Sizescited by 147
  9. 2020On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Themcited by 59
  10. 2019A New Defense Against Adversarial Images: Turning a Weakness into a Strengthcited by 49
  11. 2018Detection based Defense against Adversarial Examples from the Steganalysis Point of Viewcited by 32
  12. 2017Residual Convolutional CTC Networks for Automatic Speech Recognitioncited by 21
  13. 2019Understanding Adversarial Robustness Through Loss Landscape Geometriescited by 12
  14. 2016Training Recurrent Neural Networks by Diffusioncited by 9

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 0; 2024: 0; 2025: 1.

5 papers.

  1. 2022Exploring the Relationship between Architecture and Adversarially Robust Generalizationcited by 23
  2. 2022Alleviating Robust Overfitting of Adversarial Training With Consistency Regularizationcited by 4
  3. 2022Why Adversarial Training of ReLU Networks Is Difficult?cited by 4
  4. 2021Smoothness Analysis of Adversarial Trainingcited by 2
  5. 2025Sharpness-Aware Geometric Defense for Robust Out-Of-Distribution Detectioncited by 0