mlsanityData to 7 Oct 2026

MALI: A memory efficient and reverse accurate integrator for Neural ODEs

arXiv 2102.04668 ↗First posted 9 Feb 2021cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Neural ordinary differential equations (Neural ODEs) are a new family of deep-learning models with continuous depth. However, the numerical estimation of the gradient in the continuous case is not well solved: existing implementations of the adjoint method suffer from inaccuracy in reverse-time trajectory, while the naive method and the adaptive checkpoint adjoint method (ACA) have a memory cost that grows with integration time. In this project, based on the asynchronous leapfrog (ALF) solver, we propose the Memory-efficient ALF Integrator (MALI), which has a constant memory cost textitw.r.t number of solver steps in integration similar to the adjoint method, and guarantees accuracy in reverse-time trajectory (hence accuracy in gradient estimation). We validate MALI in various tasks: on image recognition tasks, to our knowledge, MALI is the first to enable feasible training of a Neural ODE on ImageNet and outperform a well-tuned ResNet, while existing methods fail due to either heavy memory burden or inaccuracy; for time series modeling, MALI significantly outperforms the adjoint method; and for continuous generative models, MALI achieves new state-of-the-art performance. We provide a pypi package at urlhttps://jzkay12.github.io/TorchDiffEqPack/

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 9 Feb 2021.

Builds on

28 of 45 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2016Density estimation using Real NVPcited by 2,029
  5. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  6. 2018DeepMind Control Suitecited by 935
  7. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
  8. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  9. 2019Augmented Neural ODEscited by 396
  10. 2018Invertible Residual Networkscited by 364
  11. 2017Stable Architectures for Deep Neural Networkscited by 340
  12. 2020Scalable Gradients for Stochastic Differential Equationscited by 264
16 more
  1. 2019Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Designcited by 258
  2. 2019Residual Flows for Invertible Generative Modelingcited by 256
  3. 2020How to train your neural ODE: the world of Jacobian and kinetic regularizationcited by 228
  4. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  5. 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
  6. 2019Symplectic ODE-Net: Learning Hamiltonian Dynamics with Controlcited by 183
  7. 2020Dissecting Neural ODEscited by 150
  8. 2019Neural Jump Stochastic Differential Equationscited by 147
  9. 2019Hamiltonian Graph Networks with ODE Integratorscited by 107
  10. 2019A Machine Learning Framework for Solving High-Dimensional Mean Field Game and Mean Field Control Problemscited by 95
  11. 2019Unconstrained Monotonic Neural Networkscited by 91
  12. 2019On Robustness of Neural Ordinary Differential Equationscited by 83
  13. 2020Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODEcited by 76
  14. 2020Continuous-in-Depth Neural Networkscited by 40
  15. 2019SNODE: Spectral Discretization of Neural ODEs for System Identificationcited by 26
  16. 2020Interpolation Technique to Speed Up Gradients Propagation in Neural ODEscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 8; 2023: 6; 2024: 7; 2025: 6; 2026: 2.

42 papers.

Sort:

By area:

  1. 2024Normalizing Flows are Capable Generative Modelscited by 89
  2. 2021Efficient and Accurate Gradients for Neural SDEscited by 68
  3. 2021Machine learning structure preserving brackets for forecasting irreversible processescited by 51
  4. 2021Neural Controlled Differential Equations for Online Prediction Taskscited by 46
  5. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  6. 2021Attentive Neural Controlled Differential Equations for Time-series Classification and Forecastingcited by 28
  7. 2022Do Residual Neural Networks discretize Neural Ordinary Differential Equations?cited by 27
  8. 2021Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristicscited by 26
  9. 2022EXIT: Extrapolation and Interpolation-based Neural Controlled Differential Equations for Time-series Classification and Forecastingcited by 22
  10. 2021Variational multiple shooting for Bayesian ODEs with Gaussian processescited by 15
  11. 2021Symplectic Adjoint Method for Exact Gradient of Neural ODE with Minimal Memorycited by 11
  12. 2021Second-Order Neural ODE Optimizercited by 11
30 more
  1. 2021How Does Momentum Benefit Deep Neural Networks Architecture Design? A Few Case Studiescited by 10
  2. 2024Efficient, Accurate and Stable Gradients for Neural ODEscited by 9
  3. 2022TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speedcited by 8
  4. 2024Simulation-Free Training of Neural ODEs on Paired Datacited by 8
  5. 2025Flexible Language Modeling in Continuous Space with Transformer-based Autoregressive Flowscited by 8
  6. 2022Closed-Form Diffeomorphic Transformations for Time Series Alignmentcited by 7
  7. 2023PGODE: Towards High-quality System Dynamics Modelingcited by 7
  8. 2023Embracing the chaos: analysis and diagnosis of numerical instability in variational flowscited by 6
  9. 2025Reversible Deep Equilibrium Modelscited by 5
  10. 2022Learning Subgrid-scale Models with Neural Ordinary Differential Equationscited by 4
  11. 2022A memory-efficient neural ODE framework based on high-level adjoint differentiationcited by 3
  12. 2023Faster Training of Neural ODEs Using Gauss-Legendre Quadraturecited by 3
  13. 2021Learning ODEs via Diffeomorphisms for Fast and Robust Integrationcited by 2
  14. 2024Semi-Implicit Neural Ordinary Differential Equationscited by 2
  15. 2025Latent Stochastic Interpolantscited by 2
  16. 2021Distributional Gradient Matching for Learning Uncertain Neural Dynamics Modelscited by 1
  17. 2021Characteristic Neural Ordinary Differential Equationscited by 1
  18. 2022Prediction-based One-shot Dynamic Parking Pricingcited by 1
  19. 2022Semi-Equivariant Continuous Normalizing Flows for Target-Aware Molecule Generationcited by 1
  20. 2023Long-term Time Series Forecasting based on Decomposition and Neural Ordinary Differential Equationscited by 1
  21. 2024Divide And Conquer: Learning Chaotic Dynamical Systems With Multistep Penalty Neural Ordinary Differential Equationscited by 1
  22. 2025Explicit and Effectively Symmetric Schemes for Neural SDEs on Lie Groupscited by 1
  23. 2025Contrastive Diffusion Alignment: Learning Structured Latents for Controllable Generationcited by 1
  24. 2023Semi-Equivariant Conditional Normalizing Flowscited by 0
  25. 2023Critical Sampling for Robust Evolution Operator Learning of Unknown Dynamical Systemscited by 0
  26. 2024DFM: Interpolant-free Dual Flow Matchingcited by 0
  27. 2024Data-Driven Uncertainty-Aware Forecasting of Sea Ice Conditions in the Gulf of Ob Based on Satellite Radar Imagerycited by 0
  28. 2025VNODE: A Piecewise Continuous Volterra Neural Networkcited by 0
  29. 2026Stable and Near-Reversible Diffusion ODE Solvers for Image Editingcited by 0
  30. 2026GRADSOLVE: fast exact gradients for ODE ensembles on GPUscited by 0