mlsanityData to 7 Oct 2026

Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent

arXiv 2207.04036 ↗First posted 8 Jul 2022cs.LG math.OCcited by 38

Not on the map. Why?

Abstract (arXiv)

As part of the effort to understand implicit bias of gradient descent in overparametrized models, several results have shown how the training trajectory on the overparametrized model can be understood as mirror descent on a different objective. The main result here is a characterization of this phenomenon under a notion termed commuting parametrization, which encompasses all the previous results in this setting. It is shown that gradient flow with any commuting parametrization is equivalent to continuous mirror descent with a related Legendre function. Conversely, continuous mirror descent with any Legendre function can be viewed as gradient flow with a related commuting parametrization. The latter result relies upon Nash's embedding theorem.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 43 of its 92 references that are in the map. First posted on arXiv 8 Jul 2022.

Builds on

43 of 92 references.

References · 43

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2014Convex Optimization: Algorithms and Complexitycited by 569
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  9. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  10. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  11. 2017Implicit Regularization in Matrix Factorizationcited by 318
  12. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
31 more
  1. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  2. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  3. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  4. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  5. 2020Directional convergence and alignment in deep learningcited by 172
  6. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  7. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  8. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  9. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  10. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  11. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  12. 2018Convergence of Gradient Descent on Separable Datacited by 108
  13. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  14. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  15. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  16. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  17. 2018Risk and parameter convergence of logistic regressioncited by 81
  18. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  19. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  20. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  21. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  22. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  23. 2020The Implicit Bias for Adaptive Optimization Algorithms on Homogeneous Neural Networkscited by 43
  24. 2021Fast Margin Maximization via Dual Accelerationcited by 33
  25. 2021The Benefits of Implicit Regularization from SGD in Least Squares Problemscited by 32
  26. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  27. 2018Hessian Riemannian gradient flows in convex programmingcited by 25
  28. 2021Understanding Deflation Process in Over-parametrized Tensor Decompositioncited by 20
  29. 2020Mirrorless Mirror Descent: A Natural Derivation of Mirror Descentcited by 18
  30. 2019The Implicit Bias of AdaGrad on Separable Datacited by 15
  31. 2022Non-convex online learning via algorithmic equivalencecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 10; 2025: 7; 2026: 12.

38 papers.

Sort:

By area:

  1. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
  2. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  3. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  4. 2024Implicit Regularization of Gradient Flow on One-Layer Softmax Attentioncited by 16
  5. 2024Imperative Learning: A Self-supervised Neuro-Symbolic Learning Framework for Robot Autonomycited by 16
  6. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 13
  7. 2024Mask in the Mirror: Implicit Sparsificationcited by 12
  8. 2023mSAM: Micro-Batch-Averaged Sharpness-Aware Minimizationcited by 10
  9. 2025Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?cited by 10
  10. 2024Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networkscited by 9
  11. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  12. 2023Implicit Regularization for Group Sparsitycited by 8
26 more
  1. 2024Implicit Bias of Mirror Flow on Separable Datacited by 7
  2. 2025Sign-In to the Lottery: Reparameterizing Sparse Training From Scratchcited by 6
  3. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  4. 2023Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regressioncited by 4
  5. 2023Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networkscited by 4
  6. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  7. 2025Neural Graduated Assignment for Maximum Common Edge Subgraphscited by 3
  8. 2025On exploration of an interior mirror descent flow for stochastic nonconvex constrained problemcited by 3
  9. 2025Intrinsic training dynamics of deep neural networkscited by 3
  10. 2025Diagonal Linear Networks and the Lasso Regularization Pathcited by 3
  11. 2026Understanding and inverse design of implicit bias in stochastic learning: a geometric perspectivecited by 3
  12. 2024Improving Implicit Regularization of SGD with Preconditioning for Least Square Problemscited by 2
  13. 2024A Mirror Descent Perspective of Smoothed Sign Descentcited by 2
  14. 2024Optimization Insights into Deep Diagonal Linear Networkscited by 2
  15. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 2
  16. 2026Policy Regret for Embedding Model Routing: Contextual Bandits with Low-Rank Expertscited by 1
  17. 2026Incremental Learning in Mirror Flowscited by 1
  18. 2023Implicit Balancing and Regularization: Generalization and Convergence Guarantees for Overparameterized Asymmetric Matrix Sensingcited by 0
  19. 2026Gradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutionscited by 0
  20. 2026Mirror Descent on Riemannian Manifoldscited by 0
  21. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 0
  22. 2026HORST: Composing Optimizer Geometries for Sparse Transformer Trainingcited by 0
  23. 2026Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Ratecited by 0
  24. 2026Online Learning on Hidden-Convex Losses via Algorithmic Equivalence: Optimal Regret, Geometric Barrier, and Bandit Feedbackcited by 0
  25. 2026Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initializationcited by 0
  26. 2026Establishing Boundary KKT Convergence of Mirror Descent through Reparameterizationcited by 0