Geometric structure of shallow neural networks and constructive mathcal L2 cost minimization

In this paper, we approach the problem of cost (loss) minimization in underparametrized shallow ReLU networks through the explicit construction of upper bounds which appeal to the structure of classification data, without use of gradient descent. A key focus is on elucidating the geometric structure of approximate and precise minimizers. We consider an L2 cost function, input space ℝM, output space mathbb RQ with Q≤ M, and training input sample size that can be arbitrarily large. We prove an upper bound on the minimum of the cost function of order O(δP) where δP measures the signal-to-noise ratio of training data. In the special case M=Q, we explicitly determine an exact degenerate local minimum of the cost function, and show that the sharp value differs from the upper bound obtained for Q≤ M by a relative error O(δP2). The proof of the upper bound yields a constructively trained network; we show that it metrizes a particular Q-dimensional subspace in the input space mathbb RM. We comment on the characterization of the global minimum of the cost function in the given context.