mlsanityData to 7 Oct 2026

Gradient descent aligns the layers of deep linear networks

arXiv 1810.02032 ↗First posted 4 Oct 2018cs.LG math.OC stat.MLcited by 182

Not on the map. Why?

Abstract (arXiv)

This paper establishes risk convergence and asymptotic weight matrix alignment --- a form of implicit regularization --- of gradient flow and gradient descent when applied to deep linear networks on linearly separable data. In more detail, for gradient flow applied to strictly decreasing loss functions (with similar results for gradient descent with particular decreasing step sizes): (i) the risk converges to 0; (ii) the normalized i-th weight matrix asymptotically equals its rank-1 approximation uivitop; (iii) these rank-1 matrices are aligned across layers, meaning |vi+1topui|→1. In the case of the logistic loss (binary cross entropy), more can be said: the linear function induced by the network --- the product of its weight matrices --- converges to the same direction as the maximum margin solution. This last property was identified in prior work, but only under assumptions on gradient descent which here are implied by the alignment phenomenon.

Key facts. Cited by 182 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 34; 2023: 26; 2024: 33; 2025: 34; 2026: 25.

182 papers.

Sort:

By area:

  1. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  2. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  3. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  4. 2021Self-supervised Learning is More Robust to Dataset Imbalancecited by 122
  5. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  6. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  7. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  8. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  9. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  10. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  11. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  12. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 69
38 more
  1. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  2. 2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
  3. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  4. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 50
  5. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 48
  6. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  7. 2022Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learningcited by 45
  8. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 44
  9. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  10. 2021Implicit Regularization in Tensor Factorizationcited by 43
  11. 2023Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stabilitycited by 42
  12. 2024Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learningcited by 41
  13. 2025Grokking at the Edge of Numerical Stabilitycited by 41
  14. 2021Understanding the role of importance weighting for deep learningcited by 38
  15. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  16. 2022Topology-aware Generalization of Decentralized SGDcited by 37
  17. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  18. 2023(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stabilitycited by 35
  19. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  20. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  21. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  22. 2024The Implicit Bias of Adam on Separable Datacited by 32
  23. 2025Symmetry in Neural Network Parameter Spacescited by 31
  24. 2022Regression as Classification: Influence of Task Formulation on Neural Network Featurescited by 29
  25. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  26. 2021Properties of the After Kernelcited by 27
  27. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  28. 2023Understanding Certified Training with Interval Bound Propagationcited by 26
  29. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  30. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  31. 2023On the Implicit Bias of Adamcited by 25
  32. 2022Convergence of gradient descent for deep neural networkscited by 24
  33. 2023Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theorycited by 24
  34. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21
  35. 2022Bayesian Interpolation with Deep Linear Networkscited by 21
  36. 2021Understanding Deflation Process in Over-parametrized Tensor Decompositioncited by 20
  37. 2022Infinite-width limit of deep linear neural networkscited by 20
  38. 2021Towards Understanding Learning in Neural Networks with Linear Teacherscited by 19

50 most cited of 182 All 182 (JSON)