Tighter Sparse Approximation Bounds for ReLU Neural Networks

A well-known line of work (Barron, 1993; Breiman, 1993; Klusowski & Barron, 2018) provides bounds on the width n of a ReLU two-layer neural network needed to approximate a function f over the ball BR(ℝd) up to error ε, when the Fourier based quantity Cf = frac1(2π)d/2 intℝd |xi|2 |hatf(xi)| dxi is finite. More recently Ongie et al. (2019) used the Radon transform as a tool for analysis of infinite-width ReLU two-layer networks. In particular, they introduce the concept of Radon-based R-norms and show that a function defined on ℝd can be represented as an infinite-width two-layer neural network if and only if its R-norm is finite. In this work, we extend the framework of Ongie et al. (2019) and define similar Radon-based semi-norms (R, U-norms) such that a function admits an infinite-width neural network representation on a bounded open set U subseteq ℝd when its R, U-norm is finite. Building on this, we derive sparse (finite-width) neural network approximation bounds that refine those of Breiman (1993); Klusowski & Barron (2018). Finally, we show that infinite-width neural network representations on bounded open sets are not unique and study their structure, providing a functional view of mode connectivity.