Universal Approximation Properties for an ODENet and a ResNet: Mathematical Analysis and Numerical Experiments

We prove a universal approximation property (UAP) for a class of ODENet and a class of ResNet, which are simplified mathematical models for deep learning systems with skip connections. The UAP can be stated as follows. Let n and m be the dimension of input and output data, and assume m≤ n. Then we show that ODENet of width n+m with any non-polynomial continuous activation function can approximate any continuous function on a compact subset on ℝn. We also show that ResNet has the same property as the depth tends to infinity. Furthermore, we derive the gradient of a loss function explicitly with respect to a certain tuning variable. We use this to construct a learning algorithm for ODENet. To demonstrate the usefulness of this algorithm, we apply it to a regression problem, a binary classification, and a multinomial classification in MNIST.