# Geometric structure of shallow neural networks and constructive mathcal L2 cost minimization

arXiv: https://arxiv.org/abs/2309.10370 · HTML: https://mlsanity.com/paper/2309.10370/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 19 Sep 2023.

## Builds on (most cited first, 22 of 22)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Implicit Bias of Gradient Descent on Linear Convolutional Networks](https://mlsanity.com/paper/1806.00468/) (2018)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [A Geometric Analysis of Neural Collapse with Unconstrained Features](https://mlsanity.com/paper/2105.02375/) (2021)
- [Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit](https://mlsanity.com/paper/1902.06015/) (2019)
- [Neural collapse with unconstrained features](https://mlsanity.com/paper/2011.11619/) (2020)
- [How to Start Training: The Effect of Initialization and Architecture](https://mlsanity.com/paper/1803.01719/) (2018)
- [Extended Unconstrained Features Model for Exploring Deep Neural Collapse](https://mlsanity.com/paper/2202.08087/) (2022)
- [Phase diagram for two-layer ReLU neural networks at infinite-width limit](https://mlsanity.com/paper/2007.07497/) (2020)
- [On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers](https://mlsanity.com/paper/2012.05420/) (2020)
- [Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions](https://mlsanity.com/paper/2006.15459/) (2020)
- [Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data](https://mlsanity.com/paper/2309.09725/) (2023)
- [Pure and Spurious Critical Points: a Geometric Study of Linear Networks](https://mlsanity.com/paper/1910.01671/) (2019)
- [Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems](https://mlsanity.com/paper/1905.09870/) (2019)
- [The loss landscape of deep linear neural networks: a second-order analysis](https://mlsanity.com/paper/2107.13289/) (2021)
- [Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networks](https://mlsanity.com/paper/2006.07356/) (2020)
- [Which Minimizer Does My Neural Network Converge To?](https://mlsanity.com/paper/2011.02408/) (2020)

## Built on by (most cited first, 4 of 4)

- [Interpretable global minima of deep ReLU neural networks on sequentially separable data](https://mlsanity.com/paper/2405.07098/) (2024)
- [Global L2 minimization at uniform exponential rate via geometrically adapted gradient descent in Deep Learning](https://mlsanity.com/paper/2311.15487/) (2023)
- [On non-approximability of zero loss global mathcal L2 minimizers by gradient descent in Deep Learning](https://mlsanity.com/paper/2311.07065/) (2023)
- [Derivation of effective gradient flow equations and dynamical truncation of training data in Deep Learning](https://mlsanity.com/paper/2501.07400/) (2025)

## Abstract (arXiv)

In this paper, we approach the problem of cost (loss) minimization in underparametrized shallow ReLU networks through the explicit construction of upper bounds which appeal to the structure of classification data, without use of gradient descent. A key focus is on elucidating the geometric structure of approximate and precise minimizers. We consider an L2 cost function, input space ℝM, output space mathbb RQ with Q≤ M, and training input sample size that can be arbitrarily large. We prove an upper bound on the minimum of the cost function of order O(δP) where δP measures the signal-to-noise ratio of training data. In the special case M=Q, we explicitly determine an exact degenerate local minimum of the cost function, and show that the sharp value differs from the upper bound obtained for Q≤ M by a relative error O(δP2). The proof of the upper bound yields a constructively trained network; we show that it metrizes a particular Q-dimensional subspace in the input space mathbb RM. We comment on the characterization of the global minimum of the cost function in the given context.