# Geometric structure of Deep Learning networks and construction of global mathcal L2 minimizers

arXiv: https://arxiv.org/abs/2309.10639 · HTML: https://mlsanity.com/paper/2309.10639/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 9 references that are in the map.
- First posted on arXiv 19 Sep 2023.

## Builds on (most cited first, 4 of 4)

- [How to Start Training: The Effect of Initialization and Architecture](https://mlsanity.com/paper/1803.01719/) (2018)
- [Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions](https://mlsanity.com/paper/2006.15459/) (2020)
- [Hidden symmetries of ReLU networks](https://mlsanity.com/paper/2306.06179/) (2023)
- [The loss landscape of deep linear neural networks: a second-order analysis](https://mlsanity.com/paper/2107.13289/) (2021)

## Built on by (most cited first, 6 of 6)

- [An entropy formula for the Deep Linear Network](https://mlsanity.com/paper/2509.09088/) (2025)
- [Interpretable global minima of deep ReLU neural networks on sequentially separable data](https://mlsanity.com/paper/2405.07098/) (2024)
- [On non-approximability of zero loss global mathcal L2 minimizers by gradient descent in Deep Learning](https://mlsanity.com/paper/2311.07065/) (2023)
- [Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning](https://mlsanity.com/paper/2501.07400/) (2025)
- [Architecture independent generalization bounds for overparametrized deep ReLU networks](https://mlsanity.com/paper/2504.05695/) (2025)
- [Explicit neural network classifiers for non-separable data](https://mlsanity.com/paper/2504.18710/) (2025)

## Abstract (arXiv)

In this paper, we explicitly determine local and global minimizers of the L2 cost function in underparametrized Deep Learning (DL) networks; our main goal is to shed light on their geometric structure and properties. We accomplish this by a direct construction, without invoking the gradient descent flow at any point of this work. We specifically consider L hidden layers, a ReLU ramp activation function, an L2 Schatten class (or Hilbert-Schmidt) cost function, input and output spaces ℝQ with equal dimension Q≥1, and hidden layers also defined on ℝQ; the training inputs are assumed to be sufficiently clustered. The training input size N can be arbitrarily large - thus, we are considering the underparametrized regime. More general settings are left to future work. We construct an explicit family of minimizers for the global minimum of the cost function in the case L≥ Q, which we show to be degenerate. Moreover, we determine a set of 2Q-1 distinct degenerate local minima of the cost function. In the context presented here, the concatenation of hidden layers of the DL network is reinterpreted as a recursive application of a em truncation map which "curates" the training inputs by minimizing their noise to signal ratio.