mlsanityData to 7 Oct 2026

Global Convergence of Gradient Descent for Deep Linear Residual Networks

arXiv 1911.00645 ↗First posted 2 Nov 2019cs.LG stat.MLcited by 16

Not on the map. Why?

Abstract (arXiv)

We analyze the global convergence of gradient descent for deep linear residual networks by proposing a new initialization: zero-asymmetric (ZAS) initialization. It is motivated by avoiding stable manifolds of saddle points. We prove that under the ZAS initialization, for an arbitrary target matrix, gradient descent converges to an varepsilon-optimal point in O(L3 log(1/varepsilon)) iterations, which scales polynomially with the network depth L. Our result and the exp(Omega(L)) convergence time for the standard initialization (Xavier or near-identity) [Shamir, 2018] together demonstrate the importance of the residual structure and the initialization in the optimization for deep linear neural networks, especially when L is large.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 2; 2023: 4; 2024: 2; 2025: 2; 2026: 1.

16 papers.

Sort:

  1. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  2. 2023Understanding Certified Training with Interval Bound Propagationcited by 26
  3. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  4. 2021Convergence of gradient descent for learning linear neural networkscited by 22
  5. 2023Implicit regularization of deep residual networks towards neural ODEscited by 20
  6. 2023Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Modelscited by 14
  7. 2021Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networkscited by 10
  8. 2024Features are fate: a theory of transfer learning in high-dimensional regressioncited by 10
  9. 2021Speeding up Deep Model Training by Sharing Weights and Then Unsharingcited by 7
  10. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5
  11. 2026ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scalingcited by 4
  12. 2022Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networkscited by 3
4 more
  1. 2023On the Training Instability of Shuffling SGD with Batch Normalizationcited by 3
  2. 2024Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learningcited by 1
  3. 2025Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networkscited by 1
  4. 2022Global Convergence Analysis of Deep Linear Networks with A One-neuron Layercited by 0