# Convolutional Neural Networks combined with Runge-Kutta Methods

arXiv: https://arxiv.org/abs/1802.08831 · HTML: https://mlsanity.com/paper/1802.08831/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Feb 2018.

## Built on by (most cited first, 18 of 18)

- [Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems](https://mlsanity.com/paper/2109.15142/) (2021)
- [LT-OCF: Learnable-Time ODE-based Collaborative Filtering](https://mlsanity.com/paper/2108.06208/) (2021)
- [Connections between Numerical Algorithms for PDEs and Neural Networks](https://mlsanity.com/paper/2107.14742/) (2021)
- [LightMove: A Lightweight Next-POI Recommendation for Taxicab Rooftop Advertising](https://mlsanity.com/paper/2108.04993/) (2021)
- [Understanding Self-attention Mechanism via Dynamical System Perspective](https://mlsanity.com/paper/2308.09939/) (2023)
- [Semantic Segmentation using Neural Ordinary Differential Equations](https://mlsanity.com/paper/2209.08667/) (2022)
- [A Generic Shared Attention Mechanism for Various Backbone Neural Networks](https://mlsanity.com/paper/2210.16101/) (2022)
- [Graph Neural Ordinary Differential Equations-based method for Collaborative Filtering](https://mlsanity.com/paper/2311.12329/) (2023)
- [Learning to Estimate and Refine Fluid Motion with Physical Dynamics](https://mlsanity.com/paper/2206.10480/) (2022)
- [AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarios](https://mlsanity.com/paper/2302.10184/) (2023)
- [Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning](https://mlsanity.com/paper/2411.03042/) (2024)
- [Rethinking ResNets: Improved Stacking Strategies With High Order Schemes](https://mlsanity.com/paper/2103.15244/) (2021)
- [Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods](https://mlsanity.com/paper/2108.13993/) (2021)
- [TaylorSwiftNet: Taylor Driven Temporal Modeling for Swift Future Frame Prediction](https://mlsanity.com/paper/2110.14392/) (2021)
- [Deep Neural Networks Inspired by Differential Equations](https://mlsanity.com/paper/2510.09685/) (2025)
- [Zero Stability Well Predicts Performance of Convolutional Neural Networks](https://mlsanity.com/paper/2206.13100/) (2022)
- [Machine learning for advancing low-temperature plasma modeling and simulation](https://mlsanity.com/paper/2307.00131/) (2023)
- [Recent Trends in Modelling the Continuous Time Series using Deep Learning: A Survey](https://mlsanity.com/paper/2409.09106/) (2024)

## Abstract (arXiv)

A convolutional neural network can be constructed using numerical methods for solving dynamical systems, since the forward pass of the network can be regarded as a trajectory of a dynamical system. However, existing models based on numerical solvers cannot avoid the iterations of implicit methods, which makes the models inefficient at inference time. In this paper, we reinterpret the pre-activation Residual Networks (ResNets) and their variants from the dynamical systems view. We consider that the iterations of implicit Runge-Kutta methods are fused into the training of these models. Moreover, we propose a novel approach to constructing network models based on high-order Runge-Kutta methods in order to achieve higher efficiency. Our proposed models are referred to as the Runge-Kutta Convolutional Neural Networks (RKCNNs). The RKCNNs are evaluated on multiple benchmark datasets. The experimental results show that RKCNNs are vastly superior to other dynamical system network models: they achieve higher accuracy with much fewer resources. They also expand the family of network models based on numerical methods for dynamical systems.