mlsanityData to 7 Oct 2026

Implicit Regularization in Matrix Factorization

arXiv 1705.09280 ↗First posted 25 May 2017stat.ML cs.LGcited by 318

Not on the map. Why?

Abstract (arXiv)

We study implicit regularization when optimizing an underdetermined quadratic objective over a matrix X with gradient descent on a factorization of X. We conjecture and provide empirical and theoretical evidence that with small enough step sizes and initialization close enough to the origin, gradient descent on a full dimensional factorization converges to the minimum nuclear norm solution.

Key facts. Cited by 318 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 45; 2022: 50; 2023: 69; 2024: 56; 2025: 41; 2026: 57.

318 papers.

Sort:

By area:

  1. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  2. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  3. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  4. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  5. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  6. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  7. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  8. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  9. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  10. 2024Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detectioncited by 102
  11. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  12. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
38 more
  1. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  2. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  3. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  4. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  5. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  6. 2023Transformers as Support Vector Machinescited by 67
  7. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  8. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  9. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  10. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  11. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  12. 2023Non-Vacuous Generalization Bounds for Large Language Modelscited by 53
  13. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50
  14. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  15. 2021Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixturescited by 48
  16. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  17. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
  18. 2021Implicit Regularization in Tensor Factorizationcited by 43
  19. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  20. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  21. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  22. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  23. 2021SGD Generalizes Better Than GD (And Regularization Doesn't Help)cited by 38
  24. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  25. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  26. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  27. 2023Deep networks for system identification: a Surveycited by 35
  28. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  29. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  30. 2024Why Do You Grok? A Theoretical Analysis of Grokking Modular Additioncited by 34
  31. 2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
  32. 2023The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensingcited by 32
  33. 2024The Implicit Bias of Adam on Separable Datacited by 32
  34. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  35. 2022Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformerscited by 30
  36. 2023Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppressioncited by 30
  37. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  38. 2022Flat minima generalize for low-rank matrix recoverycited by 28

50 most cited of 318 All 318 (JSON)