Data to 14 Oct 2026

Width Provably Matters in Optimization for Deep Linear Neural Networks

arXiv 1901.08572 ↗First posted 24 Jan 2019cs.LG stat.MLcited by 52

Not on the map. Why?

Abstract (arXiv)

We prove that for an L-layer fully-connected linear neural network, if the width of every hidden layer is tildeOmega (L · r · dout · kappa3 ), where r and kappa are the rank and the condition number of the input data, and dout is the output dimension, then gradient descent with Gaussian random initialization converges to a global minimum at a linear rate. The number of iterations to find an ε-suboptimal solution is O(kappa log(frac1ε)). Our polynomial upper bound on the total running time for wide deep linear networks and the expleft(Omegaleft(Lright)right) lower bound for narrow deep linear neural networks [Shamir, 2018] together demonstrate that wide layers are necessary for optimizing deep models.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 14; 2023: 6; 2024: 9; 2025: 3; 2026: 3.

52 papers.

Sort:

By area:

2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
2021Score-based Generative Neural Networks for Large-Scale Optimal Transportcited by 60
2021Initialization and Regularization of Factorized Neural Layerscited by 58
2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
2022A Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networkscited by 49
2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
2021Towards Demystifying Representation Learning with Non-contrastive Self-supervisioncited by 23

38 more

2021Convergence of gradient descent for learning linear neural networkscited by 22
2022Bayesian Interpolation with Deep Linear Networkscited by 21
2022Infinite-width limit of deep linear neural networkscited by 20
2021The Connection Between Approximation, Depth Separation and Learnability in Neural Networkscited by 19
2022On Feature Learning in Neural Networks with Global Convergence Guaranteescited by 13
2023On a continuous time model of gradient descent dynamics and instability in deep learningcited by 11
2021Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networkscited by 10
2021Deep Residual Networks Learn the Geodesic Curve in the Wasserstein Spacecited by 8
2021Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equationscited by 8
2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
2021Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Spacecited by 7
2024NTK-Guided Few-Shot Class Incremental Learningcited by 7
2021Convergence and Implicit Bias of Gradient Flow on Overparametrized Linear Networkscited by 5
2021Muddling Label Regularization: Deep Learning for Tabular Datasetscited by 5
2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
2024Swing-by Dynamics in Concept Learning and Compositional Generalizationcited by 5
2026ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scalingcited by 5
2022A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networkscited by 4
2024Physics-Informed Neural Networks: Minimizing Residual Loss with Wide Networks and Effective Activationscited by 4
2025A Local Polyak-Lojasiewicz and Descent Lemma of Gradient Descent For Overparametrized Linear Modelscited by 4
2022Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networkscited by 3
2022Blessing of Nonconvexity in Deep Linear Models: Depth Flattens the Optimization Landscape Around the True Solutioncited by 3
2023Enhance Diffusion to Improve Robust Generalizationcited by 3
2024Provable Acceleration of Nesterov's Accelerated Gradient for Rectangular Matrix Factorization and Linear Neural Networkscited by 3
2025The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysiscited by 2
2026The Impact of Anisotropic Covariance Structure on the Training Dynamics and Generalization Error of Linear Networkscited by 2
2022Explicitising The Implicit Intrepretability of Deep Neural Networks Via Dualitycited by 1
2022A note on Linear Bottleneck networks and their Transition to Multilinearitycited by 1
2022A Dynamics Theory of Implicit Regularization in Deep Low-Rank Matrix Factorizationcited by 1
2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
2024Implicit Regularization via Spectral Neural Networks and Non-linear Matrix Sensingcited by 1
2024Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learningcited by 1
2025Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networkscited by 1
2026Does SGD Seek Flatness or Sharpness? An Exactly Solvable Modelcited by 1
2021Learning with Neural Tangent Kernels in Near Input Sparsity Timecited by 0
2022Global Convergence Analysis of Deep Linear Networks with A One-neuron Layercited by 0
2022Linear RNNs Provably Learn Linear Dynamic Systemscited by 0

50 most cited of 52 All 52 (JSON)