# Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting

arXiv: https://arxiv.org/abs/2109.01876 · HTML: https://mlsanity.com/paper/2109.01876/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 80 references that are in the map.
- First posted on arXiv 4 Sep 2021.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [The UEA multivariate time series classification archive, 2018](https://mlsanity.com/paper/1811.00075/) (2018)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series](https://mlsanity.com/paper/1905.12374/) (2019)
- [Recurrent Neural Network Regularization](https://mlsanity.com/paper/1409.2329/) (2014)
- [Latent ODEs for Irregularly-Sampled Time Series](https://mlsanity.com/paper/1907.03907/) (2019)
- [Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit](https://mlsanity.com/paper/1905.09883/) (2019)
- [An Attentive Survey of Attention Models](https://mlsanity.com/paper/1904.02874/) (2019)
- [Attention in Natural Language Processing](https://mlsanity.com/paper/1902.02181/) (2019)
- [Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE](https://mlsanity.com/paper/2006.02493/) (2020)
- [DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding](https://mlsanity.com/paper/1709.04696/) (2017)
- [MALI: A memory efficient and reverse accurate integrator for Neural ODEs](https://mlsanity.com/paper/2102.04668/) (2021)
- [Attention Models in Graphs: A Survey](https://mlsanity.com/paper/1807.07984/) (2018)
- [Neural Dynamics on Complex Networks](https://mlsanity.com/paper/1908.06491/) (2019)
- [Neural Jump Ordinary Differential Equations: Consistent Continuous-Time Prediction and Filtering](https://mlsanity.com/paper/2006.04727/) (2020)
- [Describing Multimedia Content using Attention-based Encoder--Decoder Networks](https://mlsanity.com/paper/1507.01053/) (2015)
- [Gated recurrent units viewed through the lens of continuous time dynamical systems](https://mlsanity.com/paper/1906.01005/) (2019)
- [Dynamic Meta-Embeddings for Improved Sentence Representations](https://mlsanity.com/paper/1804.07983/) (2018)
- [ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations](https://mlsanity.com/paper/2105.14953/) (2021)

## Built on by (most cited first, 29 of 29)

- [ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling](https://mlsanity.com/paper/2402.10635/) (2024)
- [GT-GAN: General Purpose Time Series Synthesis with Generative Adversarial Networks](https://mlsanity.com/paper/2210.02040/) (2022)
- [Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data](https://mlsanity.com/paper/2402.14989/) (2024)
- [Universal Time-Series Representation Learning: A Survey](https://mlsanity.com/paper/2401.03717/) (2024)
- [EXIT: Extrapolation and Interpolation-based Neural Controlled Differential Equations for Time-series Classification and Forecasting](https://mlsanity.com/paper/2204.08771/) (2022)
- [Comprehensive Review of Neural Differential Equations for Time Series Analysis](https://mlsanity.com/paper/2502.09885/) (2025)
- [Learnable Path in Neural Controlled Differential Equations](https://mlsanity.com/paper/2301.04333/) (2023)
- [Signed Graph Neural Ordinary Differential Equation for Modeling Continuous-time Dynamics](https://mlsanity.com/paper/2312.11198/) (2023)
- [Tracing Agentic Failure from the Flow of Success](https://mlsanity.com/paper/2607.12747/) (2026)
- [TimeKit: A Time-series Forecasting-based Upgrade Kit for Collaborative Filtering](https://mlsanity.com/paper/2211.04266/) (2022)
- [Addressing Prediction Delays in Time Series Forecasting: A Continuous GRU Approach with Derivative Regularization](https://mlsanity.com/paper/2407.01622/) (2024)
- [HD-NDEs: Neural Differential Equations for Hallucination Detection in LLMs](https://mlsanity.com/paper/2506.00088/) (2025)
- [DualDynamics: Synergizing Implicit and Explicit Methods for Robust Irregular Time Series Analysis](https://mlsanity.com/paper/2401.04979/) (2024)
- [GDFlow: Anomaly Detection with NCDE-based Normalizing Flow for Advanced Driver Assistance System](https://mlsanity.com/paper/2409.05346/) (2024)
- [Neural Predictive Control to Coordinate Discrete- and Continuous-Time Models for Time-Series Analysis with Control-Theoretical Improvements](https://mlsanity.com/paper/2508.01833/) (2025)
- [LORD: Lower-Dimensional Embedding of Log-Signature in Neural Rough Differential Equations](https://mlsanity.com/paper/2204.08781/) (2022)
- [FlowPath: Learning Data-Driven Manifolds with Invertible Flows for Robust Irregularly-sampled Time Series Classification](https://mlsanity.com/paper/2511.10841/) (2025)
- [Long-term Time Series Forecasting based on Decomposition and Neural Ordinary Differential Equations](https://mlsanity.com/paper/2311.04522/) (2023)
- [Bridging Dynamic Factor Models and Neural Controlled Differential Equations for Nowcasting GDP](https://mlsanity.com/paper/2409.08732/) (2024)
- [Continuum Dropout for Neural Differential Equations](https://mlsanity.com/paper/2511.10446/) (2025)
- [Efficient Neural Controlled Differential Equations via Attentive Kernel Smoothing](https://mlsanity.com/paper/2602.02157/) (2026)
- [Spatial Shortcuts in Graph Neural Controlled Differential Equations](https://mlsanity.com/paper/2410.19673/) (2024)
- [TANDEM: Temporal Attention-guided Neural Differential Equations for Missingness in Time Series Classification](https://mlsanity.com/paper/2508.17519/) (2025)
- [Random Controlled Differential Equations](https://mlsanity.com/paper/2512.23670/) (2025)
- [U-Former ODE: Fast Probabilistic Forecasting of Irregular Time Series](https://mlsanity.com/paper/2602.11738/) (2026)
- [INDEQS: Informed Neural controlled Differential EQuationS](https://mlsanity.com/paper/2606.19138/) (2026)
- [Continuous-Time Acoustic Modelling with Neural Controlled Differential Equations](https://mlsanity.com/paper/2609.11725/) (2026)
- [Continuous-Time Machine Learning: A Unified Mathematical Perspective](https://mlsanity.com/paper/2609.16710/) (2026)
- [LogSig-SSM: Time-Series Modelling with Multi-Scale Log-Signature Compression for State-Space Models](https://mlsanity.com/paper/2610.05051/) (2026)

## Abstract (arXiv)

Neural networks inspired by differential equations have proliferated for the past several years. Neural ordinary differential equations (NODEs) and neural controlled differential equations (NCDEs) are two representative examples of them. In theory, NCDEs provide better representation learning capability for time-series data than NODEs. In particular, it is known that NCDEs are suitable for processing irregular time-series data. Whereas NODEs have been successfully extended after adopting attention, however, it had not been studied yet how to integrate attention into NCDEs. To this end, we present the method of Attentive Neural Controlled Differential Equations (ANCDEs) for time-series classification and forecasting, where dual NCDEs are used: one for generating attention values, and the other for evolving hidden vectors for a downstream machine learning task. We conduct experiments with three real-world time-series datasets and 10 baselines. After dropping some values, we also conduct irregular time-series experiments. Our method consistently shows the best accuracy in all cases by non-trivial margins. Our visualizations also show that the presented attention mechanism works as intended by focusing on crucial information.