Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximation

The study of universal approximation properties (UAP) for neural networks (NN) has a long history. When the network width is unlimited, only a single hidden layer is sufficient for UAP. In contrast, when the depth is unlimited, the width for UAP needs to be not less than the critical width w^*min=max(dx,dy), where dx and dy are the dimensions of the input and output, respectively. Recently, citecai2022achieve shows that a leaky-ReLU NN with this critical width can achieve UAP for Lp functions on a compact domain K, i.e., the UAP for Lp(K,ℝdy). This paper examines a uniform UAP for the function class C(K,ℝdy) and gives the exact minimum width of the leaky-ReLU NN as wmin=max(dx,dy)+Delta (dx, dy), where Delta (dx, dy) is the additional dimensions for approximating continuous functions with diffeomorphisms via embedding. To obtain this result, we propose a novel lift-flow-discretization approach that shows that the uniform UAP has a deep connection with topological theory.