mlsanityData to 7 Oct 2026

Implicit Regularization in Deep Matrix Factorization

arXiv 1905.13655 ↗First posted 31 May 2019cs.LG cs.AI cs.NEcited by 402

Not on the map. Why?

Abstract (arXiv)

Efforts to understand the generalization mystery in deep learning have led to the belief that gradient-based optimization induces a form of implicit regularization, a bias towards models of low "complexity." We study the implicit regularization of gradient descent over deep linear neural networks for matrix completion and sensing, a model referred to as deep matrix factorization. Our first finding, supported by theory and experiments, is that adding depth to a matrix factorization enhances an implicit tendency towards low-rank solutions, oftentimes leading to more accurate recovery. Secondly, we present theoretical and empirical arguments questioning a nascent view by which implicit regularization in matrix factorization can be captured using simple mathematical norms. Our results point to the possibility that the language of standard regularizers may not be rich enough to fully encompass the implicit regularization brought forth by gradient-based optimization.

Key facts. Cited by 402 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 57; 2022: 70; 2023: 75; 2024: 80; 2025: 58; 2026: 62.

402 papers.

Sort:

By area:

  1. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  2. 2024The Platonic Representation Hypothesiscited by 362
  3. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  4. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
  5. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  6. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  7. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  8. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  9. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  10. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  11. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  12. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
38 more
  1. 2022Vision Transformers provably learn spatial structurecited by 92
  2. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  3. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  4. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  5. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  6. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  7. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  8. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  9. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  10. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  11. 2023Transformers as Support Vector Machinescited by 67
  12. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  13. 2023On the Foundations of Shortcut Learningcited by 64
  14. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  15. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  16. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  17. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  18. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  19. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  20. 2021Tensor Methods in Computer Vision and Deep Learningcited by 52
  21. 2023Transformers learn through gradual rank increasecited by 52
  22. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  23. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  24. 2022Rank Diminishing in Deep Neural Networkscited by 47
  25. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  26. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  27. 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 45
  28. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  29. 2021Implicit Regularization in Tensor Factorizationcited by 43
  30. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  31. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  32. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  33. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  34. 2024Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chainscited by 39
  35. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  36. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  37. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  38. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37

50 most cited of 402 All 402 (JSON)