# Neural Delay Differential Equations

arXiv: https://arxiv.org/abs/2102.10801 · HTML: https://mlsanity.com/paper/2102.10801/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 54 references that are in the map.
- First posted on arXiv 22 Feb 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [PDE-Net: Learning PDEs from Data](https://mlsanity.com/paper/1710.09668/) (2017)
- [PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network](https://mlsanity.com/paper/1812.04426/) (2018)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise](https://mlsanity.com/paper/1906.02355/) (2019)
- [Maximum Principle Based Algorithms for Deep Learning](https://mlsanity.com/paper/1710.09513/) (2017)
- [A Mean-Field Optimal Control Formulation of Deep Learning](https://mlsanity.com/paper/1807.01083/) (2018)
- [Learning to Encode Position for Transformer with Continuous Dynamical Model](https://mlsanity.com/paper/2003.09229/) (2020)
- [Multi-level Residual Networks from Dynamical Systems View](https://mlsanity.com/paper/1710.10348/) (2017)
- [The Microsoft 2017 Conversational Speech Recognition System](https://mlsanity.com/paper/1708.06073/) (2017)
- [NeuPDE: Neural Network Based Ordinary and Partial Differential Equations for Modeling Time-Dependent Data](https://mlsanity.com/paper/1908.03190/) (2019)
- [An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight Neural Networks](https://mlsanity.com/paper/1803.01299/) (2018)
- [SNODE: Spectral Discretization of Neural ODEs for System Identification](https://mlsanity.com/paper/1906.07038/) (2019)
- [Stochastic Training of Residual Networks: a Differential Equation Viewpoint](https://mlsanity.com/paper/1812.00174/) (2018)

## Built on by (most cited first, 26 of 26)

- [DRew: Dynamically Rewired Message Passing with Delay](https://mlsanity.com/paper/2305.08018/) (2023)
- [Tractable Dendritic RNNs for Reconstructing Nonlinear Dynamical Systems](https://mlsanity.com/paper/2207.02542/) (2022)
- [Neural Laplace: Learning diverse classes of differential equations in the Laplace domain](https://mlsanity.com/paper/2206.04843/) (2022)
- [Beyond Predictions in Neural ODEs: Identification and Interventions](https://mlsanity.com/paper/2106.12430/) (2021)
- [Stateful ODE-Nets using Basis Function Expansions](https://mlsanity.com/paper/2106.10820/) (2021)
- [Unveiling Delay Effects in Traffic Forecasting: A Perspective from Spatial-Temporal Delay Differential Equations](https://mlsanity.com/paper/2402.01231/) (2024)
- [From Fourier to Neural ODEs: Flow Matching for Modeling Complex Systems](https://mlsanity.com/paper/2405.11542/) (2024)
- [Learning Hamiltonian neural Koopman operator and simultaneously sustaining and discovering conservation law](https://mlsanity.com/paper/2406.02154/) (2024)
- [Gated Recurrent Neural Networks with Weighted Time-Delay Feedback](https://mlsanity.com/paper/2212.00228/) (2022)
- [Neural Piecewise-Constant Delay Differential Equations](https://mlsanity.com/paper/2201.00960/) (2022)
- [Switched Flow Matching: Eliminating Singularities via Switching ODEs](https://mlsanity.com/paper/2405.11605/) (2024)
- [Learning PDE Solution Operator for Continuous Modeling of Time-Series](https://mlsanity.com/paper/2302.00854/) (2023)
- [Learning to Decouple Complex Systems](https://mlsanity.com/paper/2302.01581/) (2023)
- [Time and State Dependent Neural Delay Differential Equations](https://mlsanity.com/paper/2306.14545/) (2023)
- [A Dynamical Systems Perspective on the Analysis of Neural Networks](https://mlsanity.com/paper/2507.05164/) (2025)
- [Symbolic Neural Ordinary Differential Equations](https://mlsanity.com/paper/2503.08059/) (2025)
- [Modeling Irregular Astronomical Time Series with Neural Stochastic Delay Differential Equations](https://mlsanity.com/paper/2508.17521/) (2025)
- [Characteristic Neural Ordinary Differential Equations](https://mlsanity.com/paper/2111.13207/) (2021)
- [Neural delay differential equations: learning non-Markovian closures for partially known dynamical systems](https://mlsanity.com/paper/2410.02843/) (2024)
- [Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization](https://mlsanity.com/paper/2506.00131/) (2025)
- [AirDDE: Multifactor Neural Delay Differential Equations for Air Quality Forecasting](https://mlsanity.com/paper/2603.17529/) (2026)
- [Learning Time Delay Systems with Neural Ordinary Differential Equations](https://mlsanity.com/paper/2206.14288/) (2022)
- [Breaking Free: Decoupling Forced Systems with Laplace Neural Networks](https://mlsanity.com/paper/2503.13158/) (2025)
- [Learning non-Markovian Dynamical Systems with Signature-based Encoders](https://mlsanity.com/paper/2509.12022/) (2025)
- [HS-FNO: History-Space Fourier Neural Operator for Non-Markovian Partial Differential Equations](https://mlsanity.com/paper/2605.09523/) (2026)
- [Forecasting Without Full Identification: Reliability and Computational Cost in Clot-Growth Simulation](https://mlsanity.com/paper/2608.08165/) (2026)

## Abstract (arXiv)

Neural Ordinary Differential Equations (NODEs), a framework of continuous-depth neural networks, have been widely applied, showing exceptional efficacy in coping with some representative datasets. Recently, an augmented framework has been successfully developed for conquering some limitations emergent in application of the original framework. Here we propose a new class of continuous-depth neural networks with delay, named as Neural Delay Differential Equations (NDDEs), and, for computing the corresponding gradients, we use the adjoint sensitivity method to obtain the delayed dynamics of the adjoint. Since the differential equations with delays are usually seen as dynamical systems of infinite dimension possessing more fruitful dynamics, the NDDEs, compared to the NODEs, own a stronger capacity of nonlinear representations. Indeed, we analytically validate that the NDDEs are of universal approximators, and further articulate an extension of the NDDEs, where the initial function of the NDDEs is supposed to satisfy ODEs. More importantly, we use several illustrative examples to demonstrate the outstanding capacities of the NDDEs and the NDDEs with ODEs' initial value. Specifically, (1) we successfully model the delayed dynamics where the trajectories in the lower-dimensional phase space could be mutually intersected, while the traditional NODEs without any argumentation are not directly applicable for such modeling, and (2) we achieve lower loss and higher accuracy not only for the data produced synthetically by complex models but also for the real-world image datasets, i.e., CIFAR10, MNIST, and SVHN. Our results on the NDDEs reveal that appropriately articulating the elements of dynamical systems into the network design is truly beneficial to promoting the network performance.