mlsanityData to 7 Oct 2026

On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent

arXiv 2102.09769 ↗First posted 19 Feb 2021cs.LGcited by 69

Not on the map. Why?

Abstract (arXiv)

Recent work has highlighted the role of initialization scale in determining the structure of the solutions that gradient methods converge to. In particular, it was shown that large initialization leads to the neural tangent kernel regime solution, whereas small initialization leads to so called "rich regimes". However, the initialization structure is richer than the overall scale alone and involves relative magnitudes of different weights and layers in the network. Here we show that these relative scales, which we refer to as initialization shape, play an important role in determining the learned model. We develop a novel technique for deriving the inductive bias of gradient-flow and use it to obtain closed-form implicit regularizers for multiple cases of interest.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 19 of its 25 references that are in the map. First posted on arXiv 19 Feb 2021.

Builds on

19 of 25 references.

References · 19

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018On Lazy Training in Differentiable Programmingcited by 671
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  4. 2017Implicit Regularization in Matrix Factorizationcited by 318
  5. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  6. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  7. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  8. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  9. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  10. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  11. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  12. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
7 more
  1. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  2. 2019Implicit Regularization for Optimal Sparse Recoverycited by 78
  3. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  4. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  5. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  6. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  7. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 16; 2023: 16; 2024: 9; 2025: 7; 2026: 12.

69 papers.

Sort:

  1. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  2. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  3. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  4. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  5. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  6. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  7. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  8. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  9. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  10. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  11. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  12. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
38 more
  1. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  2. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  3. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  4. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  5. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  6. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  7. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  8. 2022Incremental Learning in Diagonal Linear Networkscited by 27
  9. 2021Foolish Crowds Support Benign Overfittingcited by 24
  10. 2021More is Less: Inducing Sparsity via Overparameterizationcited by 22
  11. 2023Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyondcited by 20
  12. 2021The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networkscited by 18
  13. 2021A Theoretical Analysis of Fine-tuning with Linear Teacherscited by 17
  14. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  15. 2024Implicit Regularization of Gradient Flow on One-Layer Softmax Attentioncited by 16
  16. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
  17. 2023How connectivity structure shapes rich and lazy learning in neural circuitscited by 15
  18. 2021Implicit Regularization in Matrix Sensing via Mirror Descentcited by 11
  19. 2022Smooth over-parameterized solvers for non-smooth structured optimizationcited by 11
  20. 2024Fast Training of Sinusoidal Neural Fields via Scaling Initializationcited by 11
  21. 2025Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?cited by 10
  22. 2023Inductive biases of multi-task learning and finetuning: multiple regimes of feature reusecited by 9
  23. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  24. 2023A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descentcited by 8
  25. 2025Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking)cited by 8
  26. 2022Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Netscited by 7
  27. 2024Implicit Bias of Mirror Flow on Separable Datacited by 7
  28. 2023Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamicscited by 6
  29. 2026Path-conditioned training: a principled way to rescale ReLU neural networkscited by 6
  30. 2022From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descentcited by 5
  31. 2026A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuningcited by 5
  32. 2023Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regressioncited by 4
  33. 2023Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networkscited by 4
  34. 2023Implicit regularization in AI meets generalized hardness of approximation in optimization -- Sharp results for diagonal linear networkscited by 4
  35. 2025Intrinsic training dynamics of deep neural networkscited by 3
  36. 2025Diagonal Linear Networks and the Lasso Regularization Pathcited by 3
  37. 2023The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learningcited by 2
  38. 2023Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapultscited by 2

50 most cited of 69 All 69 (JSON)