mlsanityData to 7 Oct 2026

Training invariances and the low-rank phenomenon: beyond linear networks

arXiv 2201.11968 ↗First posted 28 Jan 2022cs.LG math.OCcited by 33

Not on the map. Why?

Abstract (arXiv)

The implicit bias induced by the training of neural networks has become a topic of rigorous study. In the limit of gradient flow and gradient descent with appropriate step size, it has been shown that when one trains a deep linear network with logistic or exponential loss on linearly separable data, the weights converge to rank-1 matrices. In this paper, we extend this theoretical result to the last few linear layers of the much wider class of nonlinear ReLU-activated feedforward networks containing fully-connected layers and skip connections. Similar to the linear case, the proof relies on specific local training invariances, sometimes referred to as alignment, which we show to hold for submatrices where neurons are stably-activated in all training examples, and it reflects empirical results in the literature. We also show this is not true in general for the full matrix of ReLU fully-connected layers. Our proof relies on a specific decomposition of the network into a multilinear function and another ReLU network whose weights are constant under a certain parameter directional convergence.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 28 Jan 2022.

Builds on

17 of 31 references.

References · 17

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Densely Connected Convolutional Networkscited by 6,691
  3. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  4. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  5. 2014The Loss Surfaces of Multilayer Networkscited by 278
  6. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  7. 2016Deep Learning without Poor Local Minimacited by 233
  8. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  9. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  10. 2020Directional convergence and alignment in deep learningcited by 172
  11. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  12. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
5 more
  1. 2018Stochastic subgradient method converges on tame functionscited by 120
  2. 2015Path-SGD: Path-Normalized Optimization in Deep Neural Networkscited by 117
  3. 2020Revealing the Structure of Deep Neural Networks via Convex Dualitycited by 54
  4. 2020A mathematical model for automatic differentiation in machine learningcited by 44
  5. 2017Geometry of Optimization and Implicit Regularization in Deep Learningcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 7; 2024: 9; 2025: 7; 2026: 5.

33 papers.

Sort:

  1. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  2. 2025Symmetry in Neural Network Parameter Spacescited by 31
  3. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  4. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  5. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  6. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
  7. 2023On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Datacited by 23
  8. 2022SGD and Weight Decay Secretly Minimize the Rank of Your Neural Networkcited by 17
  9. 2023Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compressioncited by 17
  10. 2023Bottleneck Structure in Learned Features: Low-Dimension vs Regularity Tradeoffcited by 17
  11. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  12. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
21 more
  1. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  2. 2024On the hardness of learning under symmetriescited by 13
  3. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 13
  4. 2022On the Implicit Bias Towards Minimal Depth of Deep Neural Networkscited by 12
  5. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  6. 2024Provable Emergence of Deep Neural Collapse and Low-Rank Bias in L2-Regularized Nonlinear Networkscited by 10
  7. 2023Are GATs Out of Balance?cited by 9
  8. 2023Deep ReLU Networks Have Surprisingly Simple Polytopescited by 5
  9. 2023Generator Born from Classifiercited by 4
  10. 2026NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Trainingcited by 4
  11. 2024When Are Bias-Free ReLU Networks Effectively Linear Networks?cited by 3
  12. 2025Intrinsic training dynamics of deep neural networkscited by 3
  13. 2022Gradient Descent for Low-Rank Functionscited by 2
  14. 2024Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networkscited by 2
  15. 2025GrokAlign: Geometric Characterisation and Acceleration of Grokkingcited by 2
  16. 2026The Symmetries of Three-Layer ReLU Networkscited by 2
  17. 2026Position: Weight Space Should Be a First-Class Generative AI Modalitycited by 2
  18. 2025The late-stage training dynamics of (stochastic) subgradient descent on homogeneous neural networkscited by 1
  19. 2025Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecturecited by 0
  20. 2026On the Oracle Complexity of Interpolation-Based Gradient Descentcited by 0
  21. 2026On Parameter Symmetries and Conservation Laws in Gradient Flowcited by 0