# Delay Differential Neural Networks

arXiv: https://arxiv.org/abs/2012.06800 · HTML: https://mlsanity.com/paper/2012.06800/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Dec 2020.

## Built on by (most cited first, 5 of 5)

- [DRew: Dynamically Rewired Message Passing with Delay](https://mlsanity.com/paper/2305.08018/) (2023)
- [Neural Laplace: Learning diverse classes of differential equations in the Laplace domain](https://mlsanity.com/paper/2206.04843/) (2022)
- [Learning PDE Solution Operator for Continuous Modeling of Time-Series](https://mlsanity.com/paper/2302.00854/) (2023)
- [Learning the Delay Using Neural Delay Differential Equations](https://mlsanity.com/paper/2304.01329/) (2023)
- [DDE-Find: Learning Delay Differential Equations from Noisy, Limited Data](https://mlsanity.com/paper/2405.02661/) (2024)

## Abstract (arXiv)

Neural ordinary differential equations (NODEs) treat computation of intermediate feature vectors as trajectories of ordinary differential equation parameterized by a neural network. In this paper, we propose a novel model, delay differential neural networks (DDNN), inspired by delay differential equations (DDEs). The proposed model considers the derivative of the hidden feature vector as a function of the current feature vector and past feature vectors (history). The function is modelled as a neural network and consequently, it leads to continuous depth alternatives to many recent ResNet variants. We propose two different DDNN architectures, depending on the way current and past feature vectors are considered. For training DDNNs, we provide a memory-efficient adjoint method for computing gradients and back-propagate through the network. DDNN improves the data efficiency of NODE by further reducing the number of parameters without affecting the generalization performance. Experiments conducted on synthetic and real-world image classification datasets such as Cifar10 and Cifar100 show the effectiveness of the proposed models.