# Exponential Convergence Time of Gradient Descent for One-Dimensional Deep Linear Neural Networks

arXiv: https://arxiv.org/abs/1809.08587 · HTML: https://mlsanity.com/paper/1809.08587/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2018.

## Built on by (most cited first, 22 of 22)

- [A Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networks](https://mlsanity.com/paper/2212.10701/) (2022)
- [Training Dynamics of In-Context Learning in Linear Attention](https://mlsanity.com/paper/2501.16265/) (2025)
- [Continuous vs. Discrete Optimization of Deep Neural Networks](https://mlsanity.com/paper/2107.06608/) (2021)
- [The loss landscape of deep linear neural networks: a second-order analysis](https://mlsanity.com/paper/2107.13289/) (2021)
- [Convergence of gradient descent for learning linear neural networks](https://mlsanity.com/paper/2108.02040/) (2021)
- [Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descent](https://mlsanity.com/paper/2106.13792/) (2021)
- [Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures](https://mlsanity.com/paper/2512.20607/) (2025)
- [A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with ReLU activation for piecewise linear target functions](https://mlsanity.com/paper/2108.04620/) (2021)
- [A proof of convergence for stochastic gradient descent in the training of artificial neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2104.00277/) (2021)
- [Convergence proof for stochastic gradient descent in the training of deep neural networks with ReLU activation for constant target functions](https://mlsanity.com/paper/2112.07369/) (2021)
- [Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion](https://mlsanity.com/paper/2310.02012/) (2023)
- [On the Convergence of Gradient Descent on Learning Transformers with Residual Connections](https://mlsanity.com/paper/2506.05249/) (2025)
- [Magnitude and Angle Dynamics in Training Single ReLU Neurons](https://mlsanity.com/paper/2209.13394/) (2022)
- [ANCRe: Adaptive Neural Connection Reassignment for Efficient Depth Scaling](https://mlsanity.com/paper/2602.09009/) (2026)
- [Low-Rank Adaptation Redux for Large Models](https://mlsanity.com/paper/2604.21905/) (2026)
- [Pruning at Initialization -- A Sketching Perspective](https://mlsanity.com/paper/2305.17559/) (2023)
- [Explicitising The Implicit Intrepretability of Deep Neural Networks Via Duality](https://mlsanity.com/paper/2203.16455/) (2022)
- [Convergence Analysis for Learning Orthonormal Deep Linear Neural Networks](https://mlsanity.com/paper/2311.14658/) (2023)
- [Curse of Dimensionality in Neural Network Optimization](https://mlsanity.com/paper/2502.05360/) (2025)
- [On the global convergence of gradient flow for wide shallow models beyond homogeneous nonlinearities](https://mlsanity.com/paper/2605.10775/) (2026)
- [Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer](https://mlsanity.com/paper/2201.02761/) (2022)
- [Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks](https://mlsanity.com/paper/2607.07884/) (2026)

## Abstract (arXiv)

We study the dynamics of gradient descent on objective functions of the form f(prodi=1k wi) (with respect to scalar parameters w1,ldots,wk), which arise in the context of training depth-k linear neural networks. We prove that for standard random initializations, and under mild assumptions on f, the number of iterations required for convergence scales exponentially with the depth k. We also show empirically that this phenomenon can occur in higher dimensions, where each wi is a matrix. This highlights a potential obstacle in understanding the convergence of gradient-based methods for deep linear neural networks, where k is large.