# The loss surface of deep linear networks viewed through the algebraic geometry lens

arXiv: https://arxiv.org/abs/1810.07716 · HTML: https://mlsanity.com/paper/1810.07716/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2018.

## Built on by (most cited first, 20 of 20)

- [Neural Collapse: A Review on Modelling Principles and Generalization](https://mlsanity.com/paper/2206.04041/) (2022)
- [Geometry of Linear Convolutional Networks](https://mlsanity.com/paper/2108.01538/) (2021)
- [The loss landscape of deep linear neural networks: a second-order analysis](https://mlsanity.com/paper/2107.13289/) (2021)
- [Exact Solutions of a Deep Linear Network](https://mlsanity.com/paper/2202.04777/) (2022)
- [Function Space and Critical Points of Linear Convolutional Networks](https://mlsanity.com/paper/2304.05752/) (2023)
- [Posterior Collapse of a Linear Latent Variable Model](https://mlsanity.com/paper/2205.04009/) (2022)
- [The Persistence of Neural Collapse Despite Low-Rank Bias](https://mlsanity.com/paper/2410.23169/) (2024)
- [Understanding Mode Connectivity via Parameter Space Symmetry](https://mlsanity.com/paper/2505.23681/) (2025)
- [Complex Critical Points of Deep Linear Neural Networks](https://mlsanity.com/paper/2301.12651/) (2023)
- [The Landscape of Multi-Layer Linear Neural Network From the Perspective of Algebraic Geometry](https://mlsanity.com/paper/2102.04338/) (2021)
- [A Weight Initialization Based on the Linear Product Structure for Neural Networks](https://mlsanity.com/paper/2109.00125/) (2021)
- [A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorization](https://mlsanity.com/paper/2506.20344/) (2025)
- [The Riemannian Geometry Associated to Gradient Flows of Linear Convolutional Networks](https://mlsanity.com/paper/2507.06367/) (2025)
- [Phase transitions reveal hierarchical structure in deep neural networks](https://mlsanity.com/paper/2512.11866/) (2025)
- [The Geometry of Polynomial Group Convolutional Neural Networks](https://mlsanity.com/paper/2603.29566/) (2026)
- [How regularization affects the geometry of loss functions](https://mlsanity.com/paper/2307.15744/) (2023)
- [HomOpt: A Homotopy-Based Hyperparameter Optimization Method](https://mlsanity.com/paper/2308.03317/) (2023)
- [Black holes and the loss landscape in machine learning](https://mlsanity.com/paper/2306.14817/) (2023)
- [Algebraic Approach to Ridge-Regularized Mean Squared Error Minimization in Minimal ReLU Neural Network](https://mlsanity.com/paper/2508.17783/) (2025)
- [Benign Loss Landscapes Can Coexist with Worst-Case Hardness](https://mlsanity.com/paper/2609.13057/) (2026)

## Abstract (arXiv)

By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of the deep linear neural network models. After clarifying on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous symmetries of the deep linear networks, can be straightforwardly removed by a generalized L2 regularization. Then, we establish upper bounds on the number of isolated stationary points of these networks with the help of algebraic geometry. Using these upper bounds and utilizing a numerical algebraic geometry method, we find all stationary points of modest depth and matrix size. We show that in the presence of the non-zero regularization, deep linear networks indeed possess local minima which are not the global minima. Our computational results clarify certain aspects of the loss surfaces of deep linear networks and provide novel insights.