# A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networks

arXiv: https://arxiv.org/abs/1811.03568 · HTML: https://mlsanity.com/paper/1811.03568/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2018.

## Built on by (most cited first, 8 of 8)

- [Hessian Eigenspectra of More Realistic Nonlinear Models](https://mlsanity.com/paper/2103.01519/) (2021)
- [The loss landscape of deep linear neural networks: a second-order analysis](https://mlsanity.com/paper/2107.13289/) (2021)
- [Critical Points and Convergence Analysis of Generative Deep Linear Networks Trained with Bures-Wasserstein Loss](https://mlsanity.com/paper/2303.03027/) (2023)
- [Intrinsic training dynamics of deep neural networks](https://mlsanity.com/paper/2508.07370/) (2025)
- [On the ISS Property of the Gradient Flow for Single Hidden-Layer Neural Networks with Linear Activations](https://mlsanity.com/paper/2305.09904/) (2023)
- [Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networks](https://mlsanity.com/paper/2502.11152/) (2025)
- [Computing frustration and near-monotonicity in deep neural networks](https://mlsanity.com/paper/2510.05286/) (2025)
- [On the Convergence of Overparameterized Problems: Inherent Properties of the Compositional Structure of Neural Networks](https://mlsanity.com/paper/2511.09810/) (2025)

## Abstract (arXiv)

In this paper, we propose a geometric framework to analyze the convergence properties of gradient descent trajectories in the context of linear neural networks. We translate a well-known empirical observation of linear neural nets into a conjecture that we call the overfitting conjecture which states that, for almost all training data and initial conditions, the trajectory of the corresponding gradient descent system converges to a global minimum. This would imply that the solution achieved by vanilla gradient descent algorithms is equivalent to that of the least-squares estimation, for linear neural networks of an arbitrary number of hidden layers. Built upon a key invariance property induced by the network structure, we first establish convergence of gradient descent trajectories to critical points of the square loss function in the case of linear networks of arbitrary depth. Our second result is the proof of the overfitting conjecture in the case of single-hidden-layer linear networks with an argument based on the notion of normal hyperbolicity and under a generic property on the training data (i.e., holding for almost all training data).