mlsanityData to 7 Oct 2026

Robust Implicit Regularization via Weight Normalization

arXiv 2305.05448 ↗First posted 9 May 2023cs.LG math.OCcited by 10

Not on the map. Why?

Abstract (arXiv)

Overparameterized models may have many interpolating solutions; implicit regularization refers to the hidden preference of a particular optimization method towards a certain interpolating solution among the many. A by now established line of work has shown that (stochastic) gradient descent tends to have an implicit bias towards low rank and/or sparse solutions when used to train deep linear networks, explaining to some extent why overparameterized neural network models trained by gradient descent tend to have good generalization performance in practice. However, existing theory for square-loss objectives often requires very small initialization of the trainable weights, which is at odds with the larger scale at which weights are initialized in practice for faster convergence and better generalization performance. In this paper, we aim to close this gap by incorporating and analyzing gradient flow (continuous-time version of gradient descent) with weight normalization, where the weight vector is reparameterized in terms of polar coordinates, and gradient flow is applied to the polar coordinates. By analyzing key invariants of the gradient flow and using Lojasiewicz Theorem, we show that weight normalization also has an implicit bias towards sparse solutions in the diagonal linear model, but that in contrast to plain gradient flow, weight normalization enables a robust bias that persists even if the weights are initialized at practically large scale. Experiments suggest that the gains in both convergence speed and robustness of the implicit bias are improved dramatically by using weight normalization in overparameterized diagonal linear network models.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 9 May 2023.

Builds on

34 of 46 references.

References · 34

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  6. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  7. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  8. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  9. 2017Implicit Regularization in Matrix Factorizationcited by 318
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  11. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  12. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
22 more
  1. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  2. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  3. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  4. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  5. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  6. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  7. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  8. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  9. 2021Implicit Regularization in Tensor Factorizationcited by 43
  10. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  11. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39
  12. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  13. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  14. 2020Robust Recovery via Implicit Bias of Discrepant Learning Rates for Double Over-parameterizationcited by 26
  15. 2018WNGrad: Learn the Learning Rate in Gradient Descentcited by 24
  16. 2021Convergence of gradient descent for learning linear neural networkscited by 22
  17. 2021More is Less: Inducing Sparsity via Overparameterizationcited by 22
  18. 2019Implicit Regularization and Convergence for Weight Normalizationcited by 20
  19. 2020Optimization Theory for ReLU Neural Networks Trained with Normalization Layerscited by 15
  20. 2022Deep Linear Networks for Matrix Completion -- An Infinite Depth Limitcited by 8
  21. 2020Inductive Bias of Gradient Descent for Weight Normalized Smooth Homogeneous Neural Netscited by 5
  22. 2021AdaLoss: A computationally-efficient and provably convergent adaptive gradient methodcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 1; 2026: 4.

10 papers.

  1. 2024Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysiscited by 7
  2. 2024Understanding the Generalization Benefits of Late Learning Rate Decaycited by 6
  3. 2025On the Benefits of Weight Normalization for Overparameterized Matrix Sensingcited by 3
  4. 2024Stochastic Modified Flows for Riemannian Stochastic Gradient Descentcited by 2
  5. 2024Optimization Insights into Deep Diagonal Linear Networkscited by 2
  6. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  7. 2026Normalized Architectures are Natively 4-Bitcited by 1
  8. 2023Convex optimization over a probability simplexcited by 0
  9. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 0
  10. 2026HORST: Composing Optimizer Geometries for Sparse Transformer Trainingcited by 0