mlsanityData to 7 Oct 2026

Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks

arXiv 1904.13262 ↗First posted 30 Apr 2019cs.LG math.OC stat.MLcited by 122

Not on the map. Why?

Abstract (arXiv)

When optimizing over-parameterized models, such as deep neural networks, a large set of parameters can achieve zero training error. In such cases, the choice of the optimization algorithm and its respective hyper-parameters introduces biases that will lead to convergence to specific minimizers of the objective. Consequently, this choice can be considered as an implicit regularization for the training of over-parametrized models. In this work, we push this idea further by studying the discrete gradient dynamics of the training of a two-layer linear network with the least-squares loss. Using a time rescaling, we show that, with a vanishing initialization and a small enough step size, this dynamics sequentially learns the solutions of a reduced-rank regression with a gradually increasing rank.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). First posted on arXiv 30 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 22; 2023: 24; 2024: 20; 2025: 18; 2026: 21.

122 papers.

Sort:

By area:

  1. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  2. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  3. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  4. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  5. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  6. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  7. 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
  8. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  9. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  10. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  11. 2023Transformers learn through gradual rank increasecited by 52
  12. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
38 more
  1. 2021Implicit Regularization in Tensor Factorizationcited by 43
  2. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  3. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  4. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  5. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  6. 2023Don't blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropycited by 30
  7. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  8. 2023Dynamical versus Bayesian Phase Transitions in a Toy Model of Superpositioncited by 28
  9. 2022The dynamics of representation learning in shallow, non-linear autoencoderscited by 27
  10. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  11. 2022Incremental Learning in Diagonal Linear Networkscited by 27
  12. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  13. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 25
  14. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  15. 2021More is Less: Inducing Sparsity via Overparameterizationcited by 22
  16. 2022Freeze then Train: Towards Provable Representation Learning under Spurious Correlations and Feature Noisecited by 22
  17. 2022Global Convergence of Sub-gradient Method for Robust Matrix Recovery: Small Initialization, Noisy Measurements, and Over-parameterizationcited by 21
  18. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  19. 2022Infinite-width limit of deep linear neural networkscited by 20
  20. 2023High-dimensional Asymptotics of Denoising Autoencoderscited by 19
  21. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  22. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 18
  23. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  24. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  25. 2024Implicit Regularization of Gradient Flow on One-Layer Softmax Attentioncited by 16
  26. 2025How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Datacited by 16
  27. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15
  28. 2025Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Pointscited by 15
  29. 2024Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Framecited by 14
  30. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  31. 2021Analysis of feature learning in weight-tied autoencoders via the mean field lenscited by 13
  32. 2022Provable Acceleration of Heavy Ball beyond Quadratics for a Class of Polyak-Lojasiewicz Functions when the Non-Convexity is Averaged-Outcited by 11
  33. 2023The Interpolating Information Criterion for Overparameterized Modelscited by 11
  34. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  35. 2021Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networkscited by 10
  36. 2023Robust Implicit Regularization via Weight Normalizationcited by 10
  37. 2024Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problemscited by 9
  38. 2024How Neural Networks Learn the Support is an Implicit Regularization Effect of SGDcited by 8

50 most cited of 122 All 122 (JSON)