mlsanityData to 14 Oct 2026

The Marginal Value of Adaptive Gradient Methods in Machine Learning

arXiv 1705.08292 ↗First posted 23 May 2017stat.ML cs.LGcited by 250

Not on the map. Why?

Abstract (arXiv)

Adaptive optimization methods, which perform local optimization with a metric constructed from the history of iterates, are becoming increasingly popular for training deep neural networks. Examples include AdaGrad, RMSProp, and Adam. We show that for simple overparameterized problems, adaptive methods often find drastically different solutions than gradient descent (GD) or stochastic gradient descent (SGD). We construct an illustrative binary classification problem where the data is linearly separable, GD and SGD achieve zero test error, and AdaGrad, Adam, and RMSProp attain test errors arbitrarily close to half. We additionally study the empirical generalization capability of adaptive methods on several state-of-the-art deep learning models. We observe that the solutions found by adaptive methods generalize worse (often significantly worse) than SGD, even when these solutions have better training performance. These results suggest that practitioners should reconsider the use of adaptive methods to train neural networks.

Key facts. Cited by 250 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 55; 2022: 37; 2023: 44; 2024: 44; 2025: 28; 2026: 42.

250 papers.

Sort:

By area:

  1. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 101
  2. 2021The Bayesian Learning Rulecited by 99
  3. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
  4. 2022Vision Transformers provably learn spatial structurecited by 93
  5. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  6. 2025Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs)cited by 72
  7. 2022MaskTune: Mitigating Spurious Correlations by Forcing to Explorecited by 66
  8. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
  9. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  10. 2022Predicting Out-of-Distribution Error with the Projection Normcited by 50
  11. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  12. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
38 more
  1. 2022Understanding AdamW through Proximal Methods and Scale-Freenesscited by 46
  2. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  3. 2023In Search of Lost Online Test-time Adaptation: A Surveycited by 44
  4. 2021SUPER-ADAM: Faster and Universal Framework of Adaptive Gradientscited by 40
  5. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  6. 2025In Search of Adam's Secret Saucecited by 40
  7. 2022Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscapecited by 39
  8. 2021Effective Evaluation of Deep Active Learning on Image Classification Taskscited by 37
  9. 2021Spectral Bias in Practice: The Role of Function Frequency in Generalizationcited by 37
  10. 2023Enhancing Sharpness-Aware Optimization Through Variance Suppressioncited by 37
  11. 2024The Implicit Bias of Adam on Separable Datacited by 36
  12. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  13. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  14. 2022One-Pixel Shortcut: on the Learning Preference of Deep Neural Networkscited by 29
  15. 2021Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimizationcited by 28
  16. 2023DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)cited by 28
  17. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  18. 2023On the Implicit Bias of Adamcited by 27
  19. 2022Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficientlycited by 26
  20. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 26
  21. 2024Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspectivecited by 25
  22. 2021Learning by Turning: Neural Architecture Aware Optimisationcited by 23
  23. 2022Bayesian Learning for Neural Networks: an algorithmic surveycited by 23
  24. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 22
  25. 2021Hyperparameter Optimization Is Deceiving Us, and How to Stop Itcited by 20
  26. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 20
  27. 2023Principled Weight Initialization for Hypernetworkscited by 19
  28. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 18
  29. 2021Structured Sparsity Inducing Adaptive Optimizers for Deep Learningcited by 17
  30. 2022Visualizing high-dimensional loss landscapes with Hessian directionscited by 16
  31. 2023On the Role of Server Momentum in Federated Learningcited by 16
  32. 2024Curvature-Informed SGD via General Purpose Lie-Group Preconditionerscited by 16
  33. 2024JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resourcescited by 15
  34. 2025Is your batch size the problem? Revisiting the Adam-SGD gap in language modelingcited by 15
  35. 2023A Unified Approach to Controlling Implicit Regularization via Mirror Descentcited by 14
  36. 2024AdaFisher: Adaptive Second Order Optimization via Fisher Informationcited by 14
  37. 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
  38. 2026Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentumcited by 14

50 most cited of 250 All 250 (JSON)