# Best k-layer neural network approximations

arXiv: https://arxiv.org/abs/1907.01507 · HTML: https://mlsanity.com/paper/1907.01507/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jul 2019.

## Built on by (most cited first, 4 of 4)

- [On the existence of minimizers in shallow residual ReLU neural network optimization landscapes](https://mlsanity.com/paper/2302.14690/) (2023)
- [On the existence of optimal shallow feedforward networks with ReLU activation](https://mlsanity.com/paper/2303.03950/) (2023)
- [Flat Channels to Infinity in Neural Loss Landscapes](https://mlsanity.com/paper/2506.14951/) (2025)
- [Constraining the outputs of ReLU neural networks](https://mlsanity.com/paper/2508.03867/) (2025)

## Abstract (arXiv)

We show that the empirical risk minimization (ERM) problem for neural networks has no solution in general. Given a training set s1, dots, sn ∈ ℝp with corresponding responses t1,dots,tn ∈ ℝq, fitting a k-layer neural network nuθ : ℝp → ℝq involves estimation of the weights θ ∈ ℝm via an ERM: [ infθ ∈ ℝm ; sumi=1n lVert ti - nuθ(si) rVert22. ] We show that even for k = 2, this infimum is not attainable in general for common activations like ReLU, hyperbolic tangent, and sigmoid functions. A high-level explanation is like that for the nonexistence of best rank-r approximations of higher-order tensors --- the set of parameters is not a closed set --- but the geometry involved for best k-layer neural networks approximations is more subtle. In addition, we show that for smooth activations σ(x)= 1/bigl(1 + exp(-x)bigr) and σ(x)=tanh(x), such failure to attain an infimum can happen on a positive-measured subset of responses. For the ReLU activation σ(x)=max(0,x), we completely classifying cases where the ERM for a best two-layer neural network approximation attains its infimum. As an aside, we obtain a precise description of the geometry of the space of two-layer neural networks with d neurons in the hidden layer: it is the join locus of a line and the d-secant locus of a cone.