mlsanityData to 7 Oct 2026

Implicit Regularization Towards Rank Minimization in ReLU Networks

arXiv 2201.12760 ↗First posted 30 Jan 2022cs.LG stat.MLcited by 53

Not on the map. Why?

Abstract (arXiv)

We study the conjectured relationship between the implicit regularization in neural networks, trained with gradient-based methods, and rank minimization of their weight matrices. Previously, it was proved that for linear networks (of depth 2 and vector-valued outputs), gradient flow (GF) w.r.t. the square loss acts as a rank minimization heuristic. However, understanding to what extent this generalizes to nonlinear networks is an open problem. In this paper, we focus on nonlinear ReLU networks, providing several new positive and negative results. On the negative side, we prove (and demonstrate empirically) that, unlike the linear case, GF on ReLU networks may no longer tend to minimize ranks, in a rather strong sense (even approximately, for "most" datasets of size 2). On the positive side, we reveal that ReLU networks of sufficient depth are provably biased towards low-rank solutions in several reasonable settings.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 30 Jan 2022.

Builds on

44 of 58 references.

References · 44

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2017Exploring Generalization in Deep Learningcited by 595
  4. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  5. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  6. 2017Implicit Regularization in Matrix Factorizationcited by 318
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  8. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  9. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  11. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  12. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
32 more
  1. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  2. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  3. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  4. 2020Directional convergence and alignment in deep learningcited by 172
  5. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  6. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  7. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  8. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  9. 2018Convergence of Gradient Descent on Separable Datacited by 108
  10. 2019How do infinite width bounded norm networks look in function space?cited by 105
  11. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 96
  12. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  13. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  14. 2018Risk and parameter convergence of logistic regressioncited by 81
  15. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  16. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  17. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
  18. 2020Gradient descent follows the regularization path for general lossescited by 57
  19. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  20. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  21. 2020Convex Geometry and Duality of Over-parameterized Neural Networkscited by 49
  22. 2017Compression-aware Training of Deep Networkscited by 46
  23. 2019Gradient Dynamics of Shallow Univariate ReLU Networkscited by 46
  24. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  25. 2021Implicit Regularization in Tensor Factorizationcited by 43
  26. 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41
  27. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  28. 2020Gradient Methods Never Overfit On Separable Datacited by 29
  29. 2021Towards Understanding Learning in Neural Networks with Linear Teacherscited by 19
  30. 2021Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Normcited by 18
  31. 2020On implicit regularization: Morse functions and applications to matrix factorizationcited by 11
  32. 2020Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 12; 2024: 14; 2025: 11; 2026: 7.

53 papers.

Sort:

  1. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  2. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  3. 2023Transformers as Support Vector Machinescited by 67
  4. 2023Transformers learn through gradual rank increasecited by 52
  5. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  6. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  7. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  8. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  9. 2024Learning Continually by Spectral Regularizationcited by 34
  10. 2023Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximizationcited by 29
  11. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  12. 2023Simplicity Bias in 1-Hidden Layer Neural Networkscited by 26
38 more
  1. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  2. 2022Understanding and Improving Transfer Learning of Deep Models via Neural Collapsecited by 24
  3. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
  4. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  5. 2023Understanding Unimodal Bias in Multimodal Deep Linear Networkscited by 18
  6. 2022SGD and Weight Decay Secretly Minimize the Rank of Your Neural Networkcited by 17
  7. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  8. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  9. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  10. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  11. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  12. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  13. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  14. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  15. 2024Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methodscited by 8
  16. 2024Learning to Compress: Local Rank and Information Compression in Deep Neural Networkscited by 5
  17. 2023Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regressioncited by 4
  18. 2023Transformed Low-Rank Parameterization Can Help Robust Generalization for Tensor Neural Networkscited by 4
  19. 2024Learning Low-Rank Feature for Thorax Disease Classificationcited by 4
  20. 2024On Generalization Bounds for Neural Networks with Low Rank Layerscited by 4
  21. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 4
  22. 2025Low Rank Gradients and Where to Find Themcited by 4
  23. 2026NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Trainingcited by 4
  24. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 3
  25. 2025Theoretical Guarantees for Low-Rank Compression of Deep Neural Networkscited by 3
  26. 2024Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networkscited by 2
  27. 2025GrokAlign: Geometric Characterisation and Acceleration of Grokkingcited by 2
  28. 2025Structured Transformations for Stable and Interpretable Neural Computationcited by 2
  29. 2025Do We Really Need Permutations? Impact of Model Width on Linear Mode Connectivitycited by 2
  30. 2025Quantifying Distributional Invariance in Causal Subgraph for IRM-Free Graph Generalizationcited by 2
  31. 2023Deep Neural Networks Tend To Extrapolate Predictablycited by 1
  32. 2026Why Deep Jacobian Spectra Separate: Depth-Induced Scaling and Singular-Vector Alignmentcited by 1
  33. 2026Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reductioncited by 1
  34. 2024On Sampling Strategies for Spectral Model Shardingcited by 0
  35. 2025Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecturecited by 0
  36. 2025Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networkscited by 0
  37. 2025Truncated Matrix Completion - An Empirical Studycited by 0
  38. 2026Emergent Low-Rank Training Dynamics in MLPs with Smooth Activationscited by 0

50 most cited of 53 All 53 (JSON)