Neural Network Approximation: Three Hidden Layers Are Enough

A three-hidden-layer neural network with super approximation power is introduced. This network is built with the floor function (lfloor xrfloor), the exponential function (2x), the step function (1x≥ 0), or their compositions as the activation function in each neuron and hence we call such networks as Floor-Exponential-Step (FLES) networks. For any width hyper-parameter N∈mathbbN^+, it is shown that FLES networks with width maxd,N and three hidden layers can uniformly approximate a H"older continuous function f on [0,1]d with an exponential approximation rate 3λ (2√d)α 2-α N, where α ∈(0,1] and λ>0 are the H"older order and constant, respectively. More generally for an arbitrary continuous function f on [0,1]d with a modulus of continuity ωf(·), the constructive approximation rate is 2ωf(2√d)2-N+ωf(2√d,2-N). Moreover, we extend such a result to general bounded continuous functions on a bounded set Esubseteqℝd. As a consequence, this new class of networks overcomes the curse of dimensionality in approximation power when the variation of ωf(r) as rrightarrow 0 is moderate (e.g., ωf(r)lesssim rα for H"older continuous functions), since the major term to be concerned in our approximation rate is essentially √d times a function of N independent of d within the modulus of continuity. Finally, we extend our analysis to derive similar approximation results in the Lp-norm for p∈[1,∞) via replacing Floor-Exponential-Step activation functions by continuous activation functions.