# "Hey, that's not an ODE": Faster ODE Adjoints via Seminorms

arXiv: https://arxiv.org/abs/2009.09457 · HTML: https://mlsanity.com/paper/2009.09457/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2020.

## Built on by (most cited first, 30 of 35)

- [Neural Flows: Efficient Alternative to Neural ODEs](https://mlsanity.com/paper/2110.13040/) (2021)
- [Closed-form Continuous-time Neural Models](https://mlsanity.com/paper/2106.13898/) (2021)
- [Neural Controlled Differential Equations for Online Prediction Tasks](https://mlsanity.com/paper/2106.11028/) (2021)
- [LyaNet: A Lyapunov Framework for Training Neural ODEs](https://mlsanity.com/paper/2202.02526/) (2022)
- [SDE Matching: Scalable and Simulation-Free Training of Latent Stochastic Differential Equations](https://mlsanity.com/paper/2502.02472/) (2025)
- [Comprehensive Review of Neural Differential Equations for Time Series Analysis](https://mlsanity.com/paper/2502.09885/) (2025)
- [Homotopy-based training of NeuralODEs for accurate dynamics discovery](https://mlsanity.com/paper/2210.01407/) (2022)
- [A Neural ODE Interpretation of Transformer Layers](https://mlsanity.com/paper/2212.06011/) (2022)
- [Accelerating Neural ODEs Using Model Order Reduction](https://mlsanity.com/paper/2105.14070/) (2021)
- [PGODE: Towards High-quality System Dynamics Modeling](https://mlsanity.com/paper/2311.06554/) (2023)
- [BoltzNCE: Learning Likelihoods for Boltzmann Generation with Stochastic Interpolants and Noise Contrastive Estimation](https://mlsanity.com/paper/2507.00846/) (2025)
- [Scalable Mechanistic Neural Networks for Differential Equations and Machine Learning](https://mlsanity.com/paper/2410.06074/) (2024)
- [AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarios](https://mlsanity.com/paper/2302.10184/) (2023)
- [DualDynamics: Synergizing Implicit and Explicit Methods for Robust Irregular Time Series Analysis](https://mlsanity.com/paper/2401.04979/) (2024)
- [On the design space between molecular mechanics and machine learning force fields](https://mlsanity.com/paper/2409.01931/) (2024)
- [Faster Training of Neural ODEs Using Gauss-Legendre Quadrature](https://mlsanity.com/paper/2308.10644/) (2023)
- [Efficient Training of Neural Fractional-Order Differential Equation via Adjoint Backpropagation](https://mlsanity.com/paper/2503.16666/) (2025)
- [A Neural RDE-based model for solving path-dependent PDEs](https://mlsanity.com/paper/2306.01123/) (2023)
- [Continuously Evolving Graph Neural Controlled Differential Equations for Traffic Forecasting](https://mlsanity.com/paper/2401.14695/) (2024)
- [The Disappearance of Timestep Embedding in Modern Time-Dependent Neural Networks](https://mlsanity.com/paper/2405.14126/) (2024)
- [Semi-Implicit Neural Ordinary Differential Equations](https://mlsanity.com/paper/2412.11301/) (2024)
- [Grow with the Flow: 4D Reconstruction of Growing Plants with Gaussian Flow Fields](https://mlsanity.com/paper/2602.08958/) (2026)
- [Latent Discretization for Continuous-time Sequence Compression](https://mlsanity.com/paper/2212.13659/) (2022)
- [Discover physical concepts and equations with machine learning](https://mlsanity.com/paper/2412.12161/) (2024)
- [Efficient Neural SDE Training using Wiener-Space Cubature](https://mlsanity.com/paper/2502.12395/) (2025)
- [Efficient Neural Controlled Differential Equations via Attentive Kernel Smoothing](https://mlsanity.com/paper/2602.02157/) (2026)
- [AirDDE: Multifactor Neural Delay Differential Equations for Air Quality Forecasting](https://mlsanity.com/paper/2603.17529/) (2026)
- [Amortized Network Intervention to Steer the Excitatory Point Processes](https://mlsanity.com/paper/2310.04159/) (2023)
- [HyBattNet: Hybrid Framework for Predicting the Remaining Useful Life of Lithium-Ion Batteries](https://mlsanity.com/paper/2505.16664/) (2025)
- [Learning non-Markovian Dynamical Systems with Signature-based Encoders](https://mlsanity.com/paper/2509.12022/) (2025)

## Abstract (arXiv)

Neural differential equations may be trained by backpropagating gradients via the adjoint method, which is another differential equation typically solved using an adaptive-step-size numerical differential equation solver. A proposed step is accepted if its error, relative to some norm, is sufficiently small; else it is rejected, the step is shrunk, and the process is repeated. Here, we demonstrate that the particular structure of the adjoint equations makes the usual choices of norm (such as L2) unnecessarily stringent. By replacing it with a more appropriate (semi)norm, fewer steps are unnecessarily rejected and the backpropagation is made faster. This requires only minor code modifications. Experiments on a wide range of tasks -- including time series, generative modeling, and physical control -- demonstrate a median improvement of 40% fewer function evaluations. On some problems we see as much as 62% fewer function evaluations, so that the overall training time is roughly halved.