Smaller generalization error derived for a deep residual neural network compared to shallow networks

Estimates of the generalization error are proved for a residual neural network with L random Fourier features layers bar zℓ+1=bar zℓ + Resumk=1Kbar bℓ ke^iωℓ kbar zℓ+ Resumk=1Kbar cℓ ke^iω'ℓ k· x. An optimal distribution for the frequencies (ωℓ k,ω'ℓ k) of the random Fourier features e^iωℓ kbar zℓ and e^iω'ℓ k· x is derived. This derivation is based on the corresponding generalization error for the approximation of the function values f(x). The generalization error turns out to be smaller than the estimate |hat f|2L1(ℝd)/(KL) of the generalization error for random Fourier features with one hidden layer and the same total number of nodes KL, in the case the L^∞-norm of f is much less than the L1-norm of its Fourier transform hat f. This understanding of an optimal distribution for random features is used to construct a new training method for a deep residual network. Promising performance of the proposed new algorithm is demonstrated in computational experiments.