# ACE-NODE: Attentive Co-Evolving Neural Ordinary Differential Equations

arXiv: https://arxiv.org/abs/2105.14953 · HTML: https://mlsanity.com/paper/2105.14953/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 46 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Hierarchical Question-Image Co-Attention for Visual Question Answering](https://mlsanity.com/paper/1606.00061/) (2016)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series](https://mlsanity.com/paper/1905.12374/) (2019)
- [How to train your neural ODE: the world of Jacobian and kinetic regularization](https://mlsanity.com/paper/2002.02798/) (2020)
- [Structured Inference Networks for Nonlinear State Space Models](https://mlsanity.com/paper/1609.09869/) (2016)
- [Deep Kalman Filters](https://mlsanity.com/paper/1511.05121/) (2015)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [An Attentive Survey of Attention Models](https://mlsanity.com/paper/1904.02874/) (2019)
- [Attention in Natural Language Processing](https://mlsanity.com/paper/1902.02181/) (2019)
- [On Robustness of Neural Ordinary Differential Equations](https://mlsanity.com/paper/1910.05513/) (2019)
- [Learning Differential Equations that are Easy to Solve](https://mlsanity.com/paper/2007.04504/) (2020)
- [Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE](https://mlsanity.com/paper/2006.02493/) (2020)
- [DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding](https://mlsanity.com/paper/1709.04696/) (2017)
- [Attention Models in Graphs: A Survey](https://mlsanity.com/paper/1807.07984/) (2018)
- [Describing Multimedia Content using Attention-based Encoder--Decoder Networks](https://mlsanity.com/paper/1507.01053/) (2015)
- [Dynamic Meta-Embeddings for Improved Sentence Representations](https://mlsanity.com/paper/1804.07983/) (2018)
- [Neural Ordinary Differential Equations for Semantic Segmentation of Individual Colon Glands](https://mlsanity.com/paper/1910.10470/) (2019)
- [Explainable Tensorized Neural Ordinary Differential Equations forArbitrary-step Time Series Prediction](https://mlsanity.com/paper/2011.13174/) (2020)

## Built on by (most cited first, 15 of 15)

- [Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data](https://mlsanity.com/paper/2402.14989/) (2024)
- [Attentive Neural Controlled Differential Equations for Time-series Classification and Forecasting](https://mlsanity.com/paper/2109.01876/) (2021)
- [LT-OCF: Learnable-Time ODE-based Collaborative Filtering](https://mlsanity.com/paper/2108.06208/) (2021)
- [Comprehensive Review of Neural Differential Equations for Time Series Analysis](https://mlsanity.com/paper/2502.09885/) (2025)
- [Unleashing The Power of Pre-Trained Language Models for Irregularly Sampled Time Series](https://mlsanity.com/paper/2408.08328/) (2024)
- [LightMove: A Lightweight Next-POI Recommendation for Taxicab Rooftop Advertising](https://mlsanity.com/paper/2108.04993/) (2021)
- [Learnable Path in Neural Controlled Differential Equations](https://mlsanity.com/paper/2301.04333/) (2023)
- [DualDynamics: Synergizing Implicit and Explicit Methods for Robust Irregular Time Series Analysis](https://mlsanity.com/paper/2401.04979/) (2024)
- [Counterfactual Probabilistic Diffusion with Expert Models](https://mlsanity.com/paper/2508.13355/) (2025)
- [Long-term Time Series Forecasting based on Decomposition and Neural Ordinary Differential Equations](https://mlsanity.com/paper/2311.04522/) (2023)
- [Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation](https://mlsanity.com/paper/2509.04154/) (2025)
- [Incorporating Interactive Facts for Stock Selection via Neural Recursive ODEs](https://mlsanity.com/paper/2210.15925/) (2022)
- [Operator-learning-inspired Modeling of Neural Ordinary Differential Equations](https://mlsanity.com/paper/2312.10274/) (2023)
- [TANDEM: Temporal Attention-guided Neural Differential Equations for Missingness in Time Series Classification](https://mlsanity.com/paper/2508.17519/) (2025)
- [Continuous-Time Machine Learning: A Unified Mathematical Perspective](https://mlsanity.com/paper/2609.16710/) (2026)

## Abstract (arXiv)

Neural ordinary differential equations (NODEs) presented a new paradigm to construct (continuous-time) neural networks. While showing several good characteristics in terms of the number of parameters and the flexibility in constructing neural networks, they also have a couple of well-known limitations: i) theoretically NODEs learn homeomorphic mapping functions only, and ii) sometimes NODEs show numerical instability in solving integral problems. To handle this, many enhancements have been proposed. To our knowledge, however, integrating attention into NODEs has been overlooked for a while. To this end, we present a novel method of attentive dual co-evolving NODE (ACE-NODE): one main NODE for a downstream machine learning task and the other for providing attention to the main NODE. Our ACE-NODE supports both pairwise and elementwise attention. In our experiments, our method outperforms existing NODE-based and non-NODE-based baselines in almost all cases by non-trivial margins.