# Accuracy and Architecture Studies of Residual Neural Network solving Ordinary Differential Equations

arXiv: https://arxiv.org/abs/2101.03583 · HTML: https://mlsanity.com/paper/2101.03583/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 52 references that are in the map.
- First posted on arXiv 10 Jan 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [DGM: A deep learning algorithm for solving partial differential equations](https://mlsanity.com/paper/1708.07469/) (2017)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network](https://mlsanity.com/paper/1812.04426/) (2018)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks](https://mlsanity.com/paper/1802.07486/) (2018)
- [VAMPnets: Deep learning of molecular kinetics](https://mlsanity.com/paper/1710.06012/) (2017)
- [Learning Deep Neural Network Representations for Koopman Operators of Nonlinear Dynamical Systems](https://mlsanity.com/paper/1708.06850/) (2017)
- [Data Driven Governing Equations Approximation Using Deep Neural Networks](https://mlsanity.com/paper/1811.05537/) (2018)
- [Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems](https://mlsanity.com/paper/1801.01236/) (2018)
- [Reversible Architectures for Arbitrarily Deep Residual Neural Networks](https://mlsanity.com/paper/1709.03698/) (2017)
- [Machine learning approximation algorithms for high-dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations](https://mlsanity.com/paper/1709.05963/) (2017)
- [NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data](https://mlsanity.com/paper/1908.03190/) (2019)
- [Deep Relaxation: partial differential equations for optimizing deep neural networks](https://mlsanity.com/paper/1704.04932/) (2017)
- [Numerical Aspects for Approximating Governing Equations Using Data](https://mlsanity.com/paper/1809.09170/) (2018)
- [Discovery of Dynamics Using Linear Multistep Methods](https://mlsanity.com/paper/1912.12728/) (2019)
- [Variational training of neural network approximations of solution maps for physical models](https://mlsanity.com/paper/1905.02789/) (2019)
- [Robust learning with implicit residual networks](https://mlsanity.com/paper/1905.10479/) (2019)
- [Learning across scales - A multiscale method for Convolution Neural Networks](https://mlsanity.com/paper/1703.02009/) (2017)

## Abstract (arXiv)

In this paper we consider utilizing a residual neural network (ResNet) to solve ordinary differential equations. Stochastic gradient descent method is applied to obtain the optimal parameter set of weights and biases of the network. We apply forward Euler, Runge-Kutta2 and Runge-Kutta4 finite difference methods to generate three sets of targets training the ResNet and carry out the target study. The well trained ResNet behaves just as its counterpart of the corresponding one-step finite difference method. In particular, we carry out (1) the architecture study in terms of number of hidden layers and neurons per layer to find the optimal ResNet structure; (2) the target study to verify the ResNet solver behaves as accurate as its finite difference method counterpart; (3) solution trajectory simulation. Even the ResNet solver looks like and is implemented in a way similar to forward Euler scheme, its accuracy can be as high as any one step method. A sequence of numerical examples are presented to demonstrate the performance of the ResNet solver.