mlsanityData to 7 Oct 2026

Implicit Regularization in Deep Learning May Not Be Explainable by Norms

arXiv 2005.06398 ↗First posted 13 May 2020cs.LG cs.NE stat.MLcited by 118

Not on the map. Why?

Abstract (arXiv)

Mathematically characterizing the implicit regularization induced by gradient-based optimization is a longstanding pursuit in the theory of deep learning. A widespread hope is that a characterization based on minimization of norms may apply, and a standard test-bed for studying this prospect is matrix factorization (matrix completion via linear neural networks). It is an open question whether norms can explain the implicit regularization in matrix factorization. The current paper resolves this open question in the negative, by proving that there exist natural matrix factorization problems on which the implicit regularization drives all norms (and quasi-norms) towards infinity. Our results suggest that, rather than perceiving the implicit regularization via norms, a potentially more useful interpretation is minimization of rank. We demonstrate empirically that this interpretation extends to a certain class of non-linear neural networks, and hypothesize that it may be key to explaining generalization in deep learning.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 26; 2022: 31; 2023: 31; 2024: 13; 2025: 8; 2026: 9.

118 papers.

Sort:

By area:

  1. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  2. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  3. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  4. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  5. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  6. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  7. 2021An Unconstrained Layer-Peeled Perspective on Neural Collapsecited by 87
  8. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  9. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  10. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  11. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  12. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
38 more
  1. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  2. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  3. 2023Transformers learn through gradual rank increasecited by 52
  4. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  5. 2021Implicit Regularization in Tensor Factorizationcited by 43
  6. 2023Maintaining Plasticity in Deep Continual Learningcited by 42
  7. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  8. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  9. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  10. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  11. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  12. 2021The Benefits of Implicit Regularization from SGD in Least Squares Problemscited by 32
  13. 2024The Implicit Bias of Adam on Separable Datacited by 32
  14. 2023Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppressioncited by 30
  15. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  16. 2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
  17. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  18. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  19. 2024Weight decay induces low-rank attention layerscited by 27
  20. 2023Simplicity Bias in 1-Hidden Layer Neural Networkscited by 26
  21. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26
  22. 2023Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descentcited by 25
  23. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  24. 2021SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochscited by 22
  25. 2021Convergence of gradient descent for learning linear neural networkscited by 22
  26. 2021More is Less: Inducing Sparsity via Overparameterizationcited by 22
  27. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  28. 2022Infinite-width limit of deep linear neural networkscited by 20
  29. 2021Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Normcited by 18
  30. 2021The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networkscited by 18
  31. 2021Implicit Bias of Linear Equivariant Networkscited by 18
  32. 2021A Theoretical Analysis of Fine-tuning with Linear Teacherscited by 17
  33. 2022Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measurescited by 17
  34. 2023Penalising the biases in norm regularisation enforces sparsitycited by 17
  35. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  36. 2024Linear Recursive Feature Machines provably recover low-rank matricescited by 16
  37. 2022The Role of Linear Layers in Nonlinear Interpolating Networkscited by 15
  38. 2023The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networkscited by 15

50 most cited of 118 All 118 (JSON)