mlsanityData to 7 Oct 2026

Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization

arXiv 2108.11371 ↗First posted 25 Aug 2021cs.LG math.OC stat.MLcited by 48

Not on the map. Why?

Abstract (arXiv)

Adaptive gradient methods such as Adam have gained increasing popularity in deep learning optimization. However, it has been observed that compared with (stochastic) gradient descent, Adam can converge to a different solution with a significantly worse test error in many deep learning applications such as image classification, even with a fine-tuned regularization. In this paper, we provide a theoretical explanation for this phenomenon: we show that in the nonconvex setting of learning over-parameterized two-layer convolutional neural networks starting from the same random initialization, for a class of data distributions (inspired from image data), Adam and gradient descent (GD) can converge to different global solutions of the training objective with provably different generalization errors, even with weight decay regularization. In contrast, we show that if the training objective is convex, and the weight decay regularization is employed, any optimization algorithms including Adam and GD will converge to the same solution if the training is successful. This suggests that the inferior generalization performance of Adam is fundamentally tied to the nonconvex landscape of deep learning optimization.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 25 Aug 2021.

Builds on

32 of 45 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2019On the Convergence of Adam and Beyondcited by 834
  4. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  5. 2018On Lazy Training in Differentiable Programmingcited by 671
  6. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  7. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  9. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  10. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  11. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  12. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
20 more
  1. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  2. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  3. 2019Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networkscited by 221
  4. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  5. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
  6. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  7. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  8. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
  9. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 125
  10. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
  11. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  12. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  13. 2020Feature Purification: How Adversarial Training Performs Robust Deep Learningcited by 107
  14. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 99
  15. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  16. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 89
  17. 2019How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?cited by 84
  18. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  19. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  20. 2016Convolutional Neural Networks Analyzed via Convolutional Sparse Codingcited by 51

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 6; 2023: 16; 2024: 11; 2025: 10; 2026: 3.

48 papers.

Sort:

By area:

  1. 2022Vision Transformers provably learn spatial structurecited by 92
  2. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 90
  3. 2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
  4. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 66
  5. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  6. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  7. 2023Robust Learning with Progressive Data Expansion Against Spurious Correlationcited by 39
  8. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  9. 2023Understanding and Improving Feature Learning for Out-of-Distribution Generalizationcited by 35
  10. 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
  11. 2024The Implicit Bias of Adam on Separable Datacited by 32
  12. 2023Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppressioncited by 30
36 more
  1. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21
  2. 2024On Convergence of Adam for Stochastic Optimization under Relaxed Assumptionscited by 21
  3. 2024Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Methodcited by 21
  4. 2023The Benefits of Mixup for Feature Learningcited by 19
  5. 2023Quantifying the Optimization and Generalization Advantages of Graph Neural Networks Over Multilayer Perceptronscited by 12
  6. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 12
  7. 2023Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representationscited by 10
  8. 2021On the One-sided Convergence of Adam-type Algorithms in Non-convex Non-concave Min-max Optimizationcited by 9
  9. 2024NLPrompt: Noise-Label Prompt Learning for Vision-Language Modelscited by 9
  10. 2025Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictionscited by 9
  11. 2023Learning to Generalize Provably in Learning to Optimizecited by 8
  12. 2023The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networkscited by 8
  13. 2023Per-Example Gradient Regularization Improves Learning Signals from Noisy Datacited by 7
  14. 2025NIRVANA: Structured Pruning Reimagined for Large Language Model Compressioncited by 6
  15. 2024Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samplescited by 5
  16. 2024Provable Benefit of Cutout and CutMix for Feature Learningcited by 4
  17. 2025On the Role of Label Noise in the Feature Learning Processcited by 4
  18. 2025How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?cited by 4
  19. 2022Empirical Study on Optimizer Selection for Out-of-Distribution Generalizationcited by 3
  20. 2023Pruning Before Training May Improve Generalization, Provablycited by 3
  21. 2023The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learningcited by 2
  22. 2023ShiftKD: Benchmarking Knowledge Distillation under Distribution Shiftcited by 2
  23. 2024Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networkscited by 2
  24. 2024Continuous-Time Analysis of Adaptive Optimization and Normalizationcited by 2
  25. 2024A Method for Enhancing Generalization of Adam by Multiple Integrationscited by 2
  26. 2025SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximationcited by 2
  27. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2
  28. 2025Implicit Bias of Per-sample Adam on Separable Data: Departure from the Full-batch Regimecited by 2
  29. 2026Muon Learns More Robust and Transferable Features than Adamcited by 1
  30. 2023Adaptive Strategies in Non-convex Optimizationcited by 0
  31. 2023Promoting Exploration in Memory-Augmented Adam using Critical Momentacited by 0
  32. 2024DeviceBERT: Applied Transfer Learning With Targeted Annotations and Vocabulary Enrichment to Identify Medical Device and Component Terminology in FDA Recall Summariescited by 0
  33. 2024GradAlign for Training-free Model Performance Inferencecited by 0
  34. 2025Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networkscited by 0
  35. 2026Same Error, Different Function: The Optimizer as an Implicit Prior in Financial Time Seriescited by 0
  36. 2026HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalizationcited by 0