mlsanityData to 7 Oct 2026

Signal Processing Meets SGD: From Momentum to Filter

arXiv 2311.02818 ↗First posted 6 Nov 2023cs.LG eess.SPcited by 0

Not on the map. Why?

Abstract (arXiv)

In deep learning, stochastic gradient descent (SGD) and its momentum-based variants are widely used for optimization. However, the internal dynamics of these methods remain underexplored. In this paper, we analyze gradient behavior through a signal processing lens, isolating key factors that influence gradient updates and revealing a critical limitation: momentum techniques lack the flexibility to adequately balance bias and variance components in gradients, resulting in gradient estimation inaccuracies. To address this issue, we introduce a novel method SGDF (SGD with Filter) based on Wiener Filter principles, which derives an optimal time-varying gain to refine gradient updates by minimizing the mean square error in gradient estimation. This method yields an optimal first-order gradient estimate, effectively balancing noise reduction and signal preservation. Furthermore, our approach could extend to adaptive optimizers, enhancing their generalization potential. Empirical results show that SGDF achieves superior convergence and generalization compared to traditional momentum methods, and performs competitively with state-of-the-art optimizers.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 59 of its 88 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

59 of 88 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2016Improved Techniques for Training GANscited by 2,938
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  9. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  11. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  12. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
38 more
  1. 2016An overview of gradient descent optimization algorithmscited by 1,021
  2. 2020On Layer Normalization in the Transformer Architecturecited by 906
  3. 2019On the Convergence of Adam and Beyondcited by 834
  4. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  5. 2017Wasserstein GANcited by 635
  6. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  7. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  8. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  9. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  10. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  11. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  12. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  13. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  14. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  15. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
  16. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  17. 2017GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibriumcited by 208
  18. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  19. 2019Adaptive Gradient Methods with Dynamic Bound of Learning Ratecited by 181
  20. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  21. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  22. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
  23. 2016Variance-based regularization with convex objectivescited by 153
  24. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
  25. 2020Momentum Improves Normalized SGDcited by 136
  26. 2017Improving Generalization Performance by Switching from Adam to SGDcited by 121
  27. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  28. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  29. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  30. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  31. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 89
  32. 2018Hessian-based Analysis of Large Batch Training and Robustness to Adversariescited by 87
  33. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  34. 2023Convergence of AdaGrad for Non-convex Objectives: Simple Proofs and Relaxed Assumptionscited by 55
  35. 2020On the distance between two neural networks and the stability of learningcited by 46
  36. 2020Adaptive Inertia: Disentangling the Effects of Adaptive Learning Rate and Momentumcited by 40
  37. 2020The Two Regimes of Deep Network Trainingcited by 30
  38. 2022Provable Adaptivity of Adam under Non-uniform Smoothnesscited by 30

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.