mlsanityData to 7 Oct 2026

Exploring Memorization in Adversarial Training

arXiv 2106.01606 ↗First posted 3 Jun 2021cs.LG cs.CV stat.MLcited by 55

Not on the map. Why?

Abstract (arXiv)

Deep learning models have a propensity for fitting the entire training set even with random labels, which requires memorization of every training sample. In this paper, we explore the memorization effect in adversarial training (AT) for promoting a deeper understanding of model capacity, convergence, generalization, and especially robust overfitting of the adversarially trained models. We first demonstrate that deep networks have sufficient capacity to memorize adversarial examples of training data with completely random labels, but not all AT algorithms can converge under the extreme circumstance. Our study of AT with random labels motivates further analyses on the convergence and generalization of AT. We find that some AT approaches suffer from a gradient instability issue and most recently suggested complexity measures cannot explain robust generalization by considering models trained on random labels. Furthermore, we identify a significant drawback of memorization in AT that it could result in robust overfitting. We then propose a new mitigation algorithm motivated by detailed memorization analyses. Extensive experiments on various datasets validate the effectiveness of the proposed method.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 69 of its 85 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

69 of 85 references.

References · 69

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2016Wide Residual Networkscited by 3,057
  8. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  9. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  10. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  11. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  12. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
38 more
  1. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  2. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  3. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  4. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  5. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  6. 2017A Closer Look at Memorization in Deep Networkscited by 936
  7. 2016Adversarial Machine Learning at Scalecited by 934
  8. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  9. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  10. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  11. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  12. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  13. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  14. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  15. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 652
  16. 2019Adversarial Training for Free!cited by 638
  17. 2017Exploring Generalization in Deep Learningcited by 595
  18. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  19. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
  20. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  21. 2020Overfitting in adversarially robust deep learningcited by 504
  22. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  23. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  24. 2017Deep Layer Aggregationcited by 436
  25. 2019Unlabeled Data Improves Adversarial Robustnesscited by 415
  26. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  27. 2019Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attackcited by 333
  28. 2017Defense against Adversarial Attacks Using High-Level Representation Guided Denoisercited by 329
  29. 2018Adversarially Robust Generalization Requires More Datacited by 318
  30. 2018Feature Denoising for Improving Adversarial Robustnesscited by 312
  31. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  32. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  33. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  34. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 275
  35. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  36. 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
  37. 2020Bag of Tricks for Adversarial Trainingcited by 199
  38. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 12; 2023: 23; 2024: 8; 2025: 7; 2026: 1.

55 papers.

Sort:

  1. 2023A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinkingcited by 85
  2. 2022Understanding Robust Overfitting of Adversarial Training and Beyondcited by 50
  3. 2022Efficient and Effective Augmentation Strategy for Adversarial Trainingcited by 43
  4. 2023Data Augmentation Alone Can Improve Adversarial Trainingcited by 43
  5. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  6. 2023Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferabilitycited by 32
  7. 2022Isometric 3D Adversarial Examples in the Physical Worldcited by 25
  8. 2022The Enemy of My Enemy is My Friend: Exploring Inverse Adversaries for Improving Adversarial Trainingcited by 22
  9. 2022Understanding and Combating Robust Overfitting via Input Loss Landscape Analysis and Regularizationcited by 22
  10. 2023On the Over-Memorization During Natural, Robust and Catastrophic Overfittingcited by 19
  11. 2023Balance, Imbalance, and Rebalance: Understanding Robust Overfitting from a Minimax Game Perspectivecited by 18
  12. 2021Label Noise in Adversarial Training: A Novel Perspective to Study Robust Overfittingcited by 14
38 more
  1. 2024Memorization in deep learning: A surveycited by 14
  2. 2023Adversarial Training Should Be Cast as a Non-Zero-Sum Gamecited by 13
  3. 2021Benign Overfitting in Adversarially Robust Linear Classificationcited by 11
  4. 2023Certified Robust Neural Networks: Generalization and Corruption Resistancecited by 11
  5. 2023Outlier Robust Adversarial Trainingcited by 10
  6. 2023Robust Spatiotemporal Traffic Forecasting with Reinforced Dynamic Adversarial Trainingcited by 9
  7. 2022One-vs-the-Rest Loss to Focus on Important Samples in Adversarial Trainingcited by 7
  8. 2023Annealing Self-Distillation Rectification Improves Adversarial Trainingcited by 7
  9. 2023AROID: Improving Adversarial Robustness Through Online Instance-Wise Data Augmentationcited by 6
  10. 2024Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Trainingcited by 6
  11. 2024Adversarial Training: A Surveycited by 6
  12. 2025Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyondcited by 6
  13. 2023On the Clean Generalization and Robust Overfitting in Adversarial Training from Two Theoretical Views: Representation Complexity and Training Dynamicscited by 5
  14. 2021Long-term Cross Adversarial Training: A Robust Meta-learning Method for Few-shot Classification Taskscited by 4
  15. 2022Alleviating Robust Overfitting of Adversarial Training With Consistency Regularizationcited by 4
  16. 2022Adversarial Training with Complementary Labels: On the Benefit of Gradually Informative Attackscited by 4
  17. 2022A3T: Accuracy Aware Adversarial Trainingcited by 4
  18. 2023Enhancing Adversarial Training via Reweighting Optimization Trajectorycited by 4
  19. 2024Exploring Adversarial Robustness of Deep State Space Modelscited by 4
  20. 2021Bridged Adversarial Trainingcited by 3
  21. 2022Strength-Adaptive Adversarial Trainingcited by 3
  22. 2023NEO-KD: Knowledge-Distillation-Based Adversarial Training for Robust Multi-Exit Neural Networkscited by 3
  23. 2024The Effectiveness of Random Forgetting for Robust Generalizationcited by 2
  24. 2024New Paradigm of Adversarial Training: Releasing Accuracy-Robustness Trade-Off via Dummy Classcited by 2
  25. 2025Understanding Adversarial Training with Energy-based Modelscited by 2
  26. 2025Failure Cases Are Better Learned But Boundary Says Sorry: Facilitating Smooth Perception Change for Accuracy-Robustness Trade-Off in Adversarial Trainingcited by 2
  27. 2023Understanding Robust Overfitting from the Feature Generalization Perspectivecited by 1
  28. 2024Can overfitted deep neural networks in adversarial training generalize? -- An approximation viewpointcited by 1
  29. 2024Adversarial Robustness Overestimation and Instability in TRADEScited by 1
  30. 2025Parameter Interpolation Adversarial Training for Robust Image Classificationcited by 1
  31. 2022Boundary Adversarial Examples Against Adversarial Overfittingcited by 0
  32. 2023PIAT: Parameter Interpolation based Adversarial Training for Image Classificationcited by 0
  33. 2023Revisiting and Advancing Adversarial Training Through A Simple Baselinecited by 0
  34. 2023Omnipotent Adversarial Training in the Wildcited by 0
  35. 2023Enhancing Robust Representation in Adversarial Training: Alignment and Exclusion Criteriacited by 0
  36. 2023Learn from the Past: A Proxy Guided Adversarial Defense Framework with Self Distillation Regularizationcited by 0
  37. 2023On robust overfitting: adversarial training induced distribution matterscited by 0
  38. 2023Rethinking Adversarial Training with Neural Tangent Kernelcited by 0

50 most cited of 55 All 55 (JSON)