mlsanityData to 7 Oct 2026

Improving Generalization Performance by Switching from Adam to SGD

arXiv 1712.07628 ↗First posted 20 Dec 2017cs.LG math.OCcited by 121

Not on the map. Why?

Abstract (arXiv)

Despite superior training outcomes, adaptive optimization methods such as Adam, Adagrad or RMSprop have been found to generalize poorly compared to Stochastic gradient descent (SGD). These methods tend to perform well in the initial portion of training but are outperformed by SGD at later stages of training. We investigate a hybrid strategy that begins training with an adaptive method and switches to SGD when appropriate. Concretely, we propose SWATS, a simple strategy which switches from Adam to SGD when a triggering condition is satisfied. The condition we propose relates to the projection of Adam steps on the gradient subspace. By design, the monitoring process for this condition adds very little overhead and does not increase the number of hyperparameters in the optimizer. We report experiments on several standard benchmarks such as: ResNet, SENet, DenseNet and PyramidNet for the CIFAR-10 and CIFAR-100 data sets, ResNet on the tiny-ImageNet data set and language modeling with recurrent networks on the PTB and WT2 data sets. The results show that our strategy is capable of closing the generalization gap between SGD and Adam on a majority of the tasks.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 12; 2023: 25; 2024: 26; 2025: 13; 2026: 21.

121 papers.

Sort:

By area:

  1. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  2. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  3. 2022A Closer Look at Smoothness in Domain Adversarial Trainingcited by 85
  4. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
  5. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 48
  6. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  7. 2021Toward Communication Efficient Adaptive Gradient Methodcited by 25
  8. 2023On the Implicit Bias of Adamcited by 25
  9. 2022WaveMix: A Resource-efficient Neural Network for Image Analysiscited by 19
  10. 2024Active Learning for Vision-Language Modelscited by 15
  11. 2021Physics-constrained Deep Learning for Robust Inverse ECG Modelingcited by 12
  12. 2021Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networkscited by 11
38 more
  1. 2021Globally Convergent Multilevel Training of Deep Residual Networkscited by 11
  2. 2022MolMiner: You only look once for chemical structure recognitioncited by 10
  3. 2023WaveMixSR: A Resource-efficient Neural Network for Image Super-resolutioncited by 10
  4. 2024Understanding Adam Requires Better Rotation Dependent Assumptionscited by 10
  5. 2024Large Kernel Distillation Network for Efficient Single Image Super-Resolutioncited by 9
  6. 2024MomentumSMoE: Integrating Momentum into Sparse Mixture of Expertscited by 9
  7. 2021Training Deep Neural Networks with Adaptive Momentum Inspired by the Quadratic Optimizationcited by 8
  8. 2023Learning to Generalize Provably in Learning to Optimizecited by 8
  9. 2025Reparameterized LLM Training via Orthogonal Equivalence Transformationcited by 8
  10. 2021Structured Dropout Variational Inference for Bayesian Neural Networkscited by 7
  11. 2023Deep Generative Models, Synthetic Tabular Data, and Differential Privacy: An Overview and Synthesiscited by 7
  12. 2024Characterizing Dynamical Stability of Stochastic Gradient Descent in Overparameterized Learningcited by 7
  13. 2024Enhancing Deep Learning with Optimized Gradient Descent: Bridging Numerical Methods and Neural Network Trainingcited by 7
  14. 2021Rethinking Adam: A Twofold Exponential Moving Average Approachcited by 6
  15. 2022Weight Expansion: A New Perspective on Dropout and Generalizationcited by 6
  16. 2023Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizerscited by 6
  17. 2024Advancing Supervised Local Learning Beyond Classification with Long-term Feature Bankcited by 6
  18. 2021TAG: Task-based Accumulated Gradients for Lifelong learningcited by 5
  19. 2022Convolutional Xformers for Visioncited by 5
  20. 2022Flatter, faster: scaling momentum for optimal speedup of SGDcited by 5
  21. 2023Adam-family Methods with Decoupled Weight Decay in Deep Learningcited by 5
  22. 2024MLAAN: Scaling Supervised Local Learning with Multilaminar Leap Augmented Auxiliary Networkcited by 5
  23. 2024HPFF: Hierarchical Locally Supervised Learning with Patch Feature Fusioncited by 5
  24. 2025On the Origin of Algorithmic Progress in AIcited by 5
  25. 2021Local Convergence of Adaptive Gradient Descent Optimizerscited by 4
  26. 2021VTLayout: Fusion of Visual and Text Features for Document Layout Analysiscited by 4
  27. 2022Optimal learning rate schedules in high-dimensional non-convex optimization problemscited by 4
  28. 2023AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrixcited by 4
  29. 2024Conformal Symplectic Optimization for Stable Reinforcement Learningcited by 4
  30. 2025High-entropy Advantage in Neural Networks' Generalizabilitycited by 4
  31. 2021Exploiting Adam-like Optimization Algorithms to Improve the Performance of Convolutional Neural Networkscited by 3
  32. 2021Quantum Enhanced Filter: QFiltercited by 3
  33. 2022An Adaptive Gradient Method with Energy and Momentumcited by 3
  34. 2024Towards Trustworthy Machine Learning in Production: An Overview of the Robustness in MLOps Approachcited by 3
  35. 2021Smoothness Analysis of Adversarial Trainingcited by 2
  36. 2021Coconut trees detection and segmentation in aerial imagery using mask region-based convolution neural networkcited by 2
  37. 2022PoF: Post-Training of Feature Extractor for Improving Generalizationcited by 2
  38. 2023Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsizecited by 2

50 most cited of 121 All 121 (JSON)