mlsanityData to 7 Oct 2026

Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networks

arXiv 1809.08587 ↗First posted 23 Sep 2018cs.LG cs.NE math.OCcited by 22

Not on the map. Why?

Abstract (arXiv)

We study the dynamics of gradient descent on objective functions of the form f(prodi=1k wi) (with respect to scalar parameters w1,ldots,wk), which arise in the context of training depth-k linear neural networks. We prove that for standard random initializations, and under mild assumptions on f, the number of iterations required for convergence scales exponentially with the depth k. We also show empirically that this phenomenon can occur in higher dimensions, where each wi is a matrix. This highlights a potential obstacle in understanding the convergence of gradient-based methods for deep linear neural networks, where k is large.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 4; 2023: 3; 2024: 0; 2025: 4; 2026: 4.

22 papers.

Sort:

  1. 2022A Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networkscited by 49
  2. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  3. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  4. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  5. 2021Convergence of gradient descent for learning linear neural networkscited by 22
  6. 2021Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descentcited by 21
  7. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  8. 2021A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functionscited by 10
  9. 2021A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functionscited by 9
  10. 2021Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functionscited by 9
  11. 2023Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosioncited by 8
  12. 2022Magnitude and Angle Dynamics in Training Single ReLU Neuronscited by 5
10 more
  1. 2025On the Convergence of Gradient Descent on Learning Transformers with Residual Connectionscited by 5
  2. 2026ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scalingcited by 4
  3. 2023Pruning at Initialization -- A Sketching Perspectivecited by 2
  4. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  5. 2022Explicitising The Implicit Intrepretability of Deep Neural Networks Via Dualitycited by 1
  6. 2023Convergence Analysis for Learning Orthonormal Deep Linear Neural Networkscited by 1
  7. 2025Curse of Dimensionality in Neural Network Optimizationcited by 1
  8. 2026On the global convergence of gradient flow for wide shallow models beyond homogeneous nonlinearitiescited by 1
  9. 2022Global Convergence Analysis of Deep Linear Networks with A One-neuron Layercited by 0
  10. 2026Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networkscited by 0