mlsanityData to 7 Oct 2026

On the Global Convergence of Training Deep Linear ResNets

arXiv 2003.01094 ↗First posted 2 Mar 2020cs.LG math.OC stat.MLcited by 29

Not on the map. Why?

Abstract (arXiv)

We study the convergence of gradient descent (GD) and stochastic gradient descent (SGD) for training L-hidden-layer linear residual networks (ResNets). We prove that for training deep residual networks with certain linear transformations at input and output layers, which are fixed throughout training, both GD and SGD with zero initialization on all hidden weights can converge to the global minimum of the training loss. Moreover, when specializing to appropriate Gaussian random linear transformations, GD and SGD provably optimize wide enough deep linear ResNets. Compared with the global convergence result of GD for training standard deep linear networks (Du & Hu 2019), our condition on the neural network width is sharper by a factor of O(kappa L), where kappa denotes the condition number of the covariance matrix of the training data. We further propose a modified identity input and output transformations, and show that a (d+k)-wide neural network is sufficient to guarantee the global convergence of GD/SGD, where d,k are the input and output dimensions respectively.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 12; 2023: 3; 2024: 3; 2025: 3; 2026: 2.

29 papers.

Sort:

  1. 2021Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depthcited by 55
  2. 2021On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layerscited by 35
  3. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  4. 2022Do Residual Neural Networks discretize Neural Ordinary Differential Equations?cited by 27
  5. 2021Convergence of gradient descent for learning linear neural networkscited by 22
  6. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
  7. 2024Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimescited by 18
  8. 2024Deep linear networks for regression are implicitly regularized towards flat minimacited by 17
  9. 2022Agnostic Learning of General ReLU Activation Using Gradient Descentcited by 12
  10. 2021Understanding Dynamics of Nonlinear Representation Learning and Its Applicationcited by 11
  11. 2021Local SGD Optimizes Overparameterized Neural Networks in Polynomial Timecited by 10
  12. 2022On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problemscited by 8
17 more
  1. 2022Deep Linear Networks can Benignly Overfit when Shallow Ones Docited by 8
  2. 2024Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transportcited by 7
  3. 2022Magnitude and Angle Dynamics in Training Single ReLU Neuronscited by 5
  4. 2023Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Spacecited by 5
  5. 2025The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learningcited by 5
  6. 2025On the Convergence of Gradient Descent on Learning Transformers with Residual Connectionscited by 5
  7. 2025COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximationcited by 5
  8. 2026Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chainingcited by 5
  9. 2022Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysiscited by 4
  10. 2026ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scalingcited by 4
  11. 2022Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networkscited by 3
  12. 2022A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networkscited by 3
  13. 2022Convergence Analysis of Deep Residual Networkscited by 1
  14. 2022Unifying Label-inputted Graph Neural Networks with Deep Equilibrium Modelscited by 1
  15. 2023Convergence Analysis for Learning Orthonormal Deep Linear Neural Networkscited by 1
  16. 2022Global Convergence Analysis of Deep Linear Networks with A One-neuron Layercited by 0
  17. 2022Linear RNNs Provably Learn Linear Dynamic Systemscited by 0