mlsanityData to 7 Oct 2026

Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functions

arXiv 2209.15055 ↗First posted 29 Sep 2022stat.ML cs.AI cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

We show that the representation cost of fully connected neural networks with homogeneous nonlinearities - which describes the implicit bias in function space of networks with L2-regularization or with losses such as the cross-entropy - converges as the depth of the network goes to infinity to a notion of rank over nonlinear functions. We then inquire under which conditions the global minima of the loss recover the `true' rank of the data: we show that for too large depths the global minimum will be approximately rank 1 (underestimating the rank); we then argue that there is a range of depths which grows with the number of datapoints where the true rank is recovered. Finally, we discuss the effect of the rank of a classifier on the topology of the resulting class boundaries and show that autoencoders with optimal nonlinear rank are naturally denoising.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

22 of 30 references.

References · 22

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  4. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  5. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  6. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  7. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  8. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  9. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  10. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  11. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  12. 2020Directional convergence and alignment in deep learningcited by 172
10 more
  1. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  2. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  3. 2019How do infinite width bounded norm networks look in function space?cited by 105
  4. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  5. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  6. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  7. 2022Rank Diminishing in Deep Neural Networkscited by 47
  8. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  9. 2022Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsitycited by 17
  10. 2022The Role of Linear Layers in Nonlinear Interpolating Networkscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 16; 2025: 10; 2026: 4.

38 papers.

Sort:

By area:

  1. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
  2. 2024Learning Continually by Spectral Regularizationcited by 34
  3. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  4. 2024Weight decay induces low-rank attention layerscited by 27
  5. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  6. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  7. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  8. 2024Linear Recursive Feature Machines provably recover low-rank matricescited by 16
  9. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  10. 2024Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapsecited by 14
  11. 2023Provable Guarantees for Neural Networks via Gradient Feature Learningcited by 13
  12. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 13
26 more
  1. 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 10
  2. 2023How do Minimum-Norm Shallow Denoisers Look in Function Space?cited by 9
  3. 2024Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learningcited by 9
  4. 2024On the Local Complexity of Linear Regions in Deep ReLU Networkscited by 9
  5. 2023ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Modelscited by 7
  6. 2025DeepCrossAttention: Supercharging Transformer Residual Connectionscited by 7
  7. 2024How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learningcited by 6
  8. 2025When Diffusion Models Memorize: Inductive Biases in Probability Flow of Minimum-Norm Shallow Neural Netscited by 6
  9. 2024Learning to Compress: Local Rank and Information Compression in Deep Neural Networkscited by 5
  10. 2025Deep Learning as a Convex Paradigm of Computation: Minimizing Circuit Size with ResNetscited by 5
  11. 2025Low-rank bias, weight decay, and model merging in neural networkscited by 4
  12. 2025Saddle-To-Saddle Dynamics in Deep ReLU Networks: Low-Rank Bias in the First Saddle Escapecited by 4
  13. 2022PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networkscited by 2
  14. 2024Depth Separation in Norm-Bounded Infinite-Width Neural Networkscited by 2
  15. 2024Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networkscited by 2
  16. 2024Generalized Probabilistic Attention Mechanism in Transformerscited by 2
  17. 2025Understanding Nonlinear Implicit Bias via Region Counts in Input Spacecited by 2
  18. 2024Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNetscited by 1
  19. 2025Conv4Rec: A 1-by-1 Convolutional AutoEncoder for User Profiling through Joint Analysis of Implicit and Explicit Feedbackscited by 1
  20. 2026Why Deep Jacobian Spectra Separate: Depth-Induced Scaling and Singular-Vector Alignmentcited by 1
  21. 2026Deep Neural Regression Collapsecited by 1
  22. 2024Local Linear Recovery Guarantee of Deep Neural Networks at Overparameterizationcited by 0
  23. 2025Explainable Neural Networks with Guarantees: A Sparse Estimation Approachcited by 0
  24. 2025Gradient Descent Robustly Learns the Intrinsic Dimension of Data in Training Convolutional Neural Networkscited by 0
  25. 2026The Implicit Bias of Depth: From Neural Collapse to Softmax Codescited by 0
  26. 2026Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networkscited by 0