A Mirror Descent Perspective of Smoothed Sign Descent
Not on the map. Why?
Abstract (arXiv)
Recent work by Woodworth et al. (2020) shows that the optimization dynamics of gradient descent for overparameterized problems can be viewed as low-dimensional dual dynamics induced by a mirror map, explaining the implicit regularization phenomenon from the mirror descent perspective. However, the methodology does not apply to algorithms where update directions deviate from true gradients, such as ADAM. We use the mirror descent framework to study the dynamics of smoothed sign descent with a stability constant varepsilon for regression problems. We propose a mirror map that establishes equivalence to dual dynamics under some assumptions. By studying dual dynamics, we characterize the convergent solution as an approximate KKT point of minimizing a Bregman divergence style function, and show the benefit of tuning the stability constant varepsilon to reduce the KKT error.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 18 Oct 2024.
Builds on
30 of 36 references.
References · 30
- 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
- 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
- 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
- 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
- 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
- 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
- 2020Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shiftcited by 236
- 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 139
- 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
18 more
- 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
- 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 103
- 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
- 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 58
- 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 46
- 2020The Geometry of Sign Gradient Descentcited by 41
- 2019Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalizationcited by 40
- 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 39
- 2020A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithmscited by 33
- 2018Stochastic Gradient/Mirror Descent: Minimax Optimality and Implicit Regularizationcited by 29
- 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
- 2022Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficientlycited by 26
- 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 22
- 2020Mirrorless Mirror Descent: A Natural Derivation of Mirror Descentcited by 18
- 2023A Unified Approach to Controlling Implicit Regularization via Mirror Descentcited by 14
- 2020EAdam Optimizer: How ε Impact Adamcited by 10
- 2023Understanding the robustness difference between stochastic gradient descent and adaptive gradient methodscited by 8
- 2020Linear Convergence of Generalized Mirror Descent with Time-Dependent Mirrorscited by 5
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.