# Operator-learning-inspired Modeling of Neural Ordinary Differential Equations

arXiv: https://arxiv.org/abs/2312.10274 · HTML: https://mlsanity.com/paper/2312.10274/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 16 Dec 2023.

## Builds on (most cited first, 26 of 26)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [U-FNO -- An enhanced Fourier neural operator-based deep-learning model for multiphase flow](https://mlsanity.com/paper/2109.03697/) (2021)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [Multiwavelet-based Operator Learning for Differential Equations](https://mlsanity.com/paper/2109.13459/) (2021)
- [U-NO: U-shaped Neural Operators](https://mlsanity.com/paper/2204.11127/) (2022)
- [How to train your neural ODE: the world of Jacobian and kinetic regularization](https://mlsanity.com/paper/2002.02798/) (2020)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Neural Rough Differential Equations for Long Time Series](https://mlsanity.com/paper/2009.08295/) (2020)
- [Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise](https://mlsanity.com/paper/1906.02355/) (2019)
- [On Second Order Behaviour in Augmented Neural ODEs](https://mlsanity.com/paper/2006.07220/) (2020)
- [Heavy Ball Neural Ordinary Differential Equations](https://mlsanity.com/paper/2110.04840/) (2021)
- [SOS: Score-based Oversampling for Tabular Data](https://mlsanity.com/paper/2206.08555/) (2022)
- [LT-OCF: Learnable-Time ODE-based Collaborative Filtering](https://mlsanity.com/paper/2108.06208/) (2021)
- [ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations](https://mlsanity.com/paper/2105.14953/) (2021)
- [AdamNODEs: When Neural ODE Meets Adaptive Moment Estimation](https://mlsanity.com/paper/2207.06066/) (2022)

## Abstract (arXiv)

Neural ordinary differential equations (NODEs), one of the most influential works of the differential equation-based deep learning, are to continuously generalize residual networks and opened a new field. They are currently utilized for various downstream tasks, e.g., image classification, time series classification, image generation, etc. Its key part is how to model the time-derivative of the hidden state, denoted dh(t)/dt. People have habitually used conventional neural network architectures, e.g., fully-connected layers followed by non-linear activations. In this paper, however, we present a neural operator-based method to define the time-derivative term. Neural operators were initially proposed to model the differential operator of partial differential equations (PDEs). Since the time-derivative of NODEs can be understood as a special type of the differential operator, our proposed method, called branched Fourier neural operator (BFNO), makes sense. In our experiments with general downstream tasks, our method significantly outperforms existing methods.