Sobolev Approximation by Fixed-Size Neural Networks with Arbitrary Accuracy

In this work, we investigate new activation functions for achieving arbitrary-accuracy Sobolev approximation by fixed-size neural networks. We first show that any function in W2,∞((a,b)d) can be approximated with arbitrary accuracy, measured in the W1,∞-norm, by a fixed-size neural network using the Elementary Universal Activation Function (EUAF). To extend this result to Ws,∞((a,b)d) for s∈mathbbN, we introduce a smooth activation DUAF∞ from the family of Differentiable Universal Activation Functions (DUAFn). We prove that any function in Ws,∞((a,b)d) can be approximated with arbitrary accuracy in the Ws-1,∞-norm by a fixed-size DUAF∞-activated network. We further construct sigmoidal variants widetildeDUAFn and show that, for every 1≤ s≤ n, fixed-size widetildeDUAFn-activated networks still approximate any f∈ Ws,∞((a,b)d) with arbitrary accuracy in the Ws-1,∞-norm. In all these results, the width and depth bounds are computed explicitly, and the proposed activations are elementary.