Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One

In this paper, we study the universal approximation property of residual neural networks. For input and output dimensions dx and dy, and LeakyReLU, ReLU, ReLU-like activation functions, the upper and lower bounds of the minimum block width are established. To achieve Lp approximation (1≤ p <+∞) on any compact set, we show that the exact minimum block width is maxdx,dy when each residual branch has inner width 1. Furthermore, we show that residual neural networks with block width mindx+dy, max2dx+1,dy can achieve uniform approximation on any compact set under the constraint that each residual branch has inner width 1. Besides, for any activation function family, we prove that there exist functions that cannot be approximated by residual neural networks with block width less than maxdx, dy, both in the Lp sense and the uniform sense, regardless of inner width. Consequently, for LeakyReLU, ReLU, ReLU-like activation functions and dy≥ 2dx+1, the exact minimum block width for uniform approximation is dy when each residual branch has inner width 1.