# How to train your neural ODE: the world of Jacobian and kinetic regularization

arXiv: https://arxiv.org/abs/2002.02798 · HTML: https://mlsanity.com/paper/2002.02798/

## Key facts

- Cited by 228 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2020.

## Built on by (most cited first, 30 of 228)

- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Building Normalizing Flows with Stochastic Interpolants](https://mlsanity.com/paper/2209.15571/) (2022)
- [Stochastic Interpolants: A Unifying Framework for Flows and Diffusions](https://mlsanity.com/paper/2303.08797/) (2023)
- [Improving and generalizing flow-based generative models with minibatch optimal transport](https://mlsanity.com/paper/2302.00482/) (2023)
- [Multisample Flow Matching: Straightening Flows with Minibatch Couplings](https://mlsanity.com/paper/2304.14772/) (2023)
- [GRAND: Graph Neural Diffusion](https://mlsanity.com/paper/2106.10934/) (2021)
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](https://mlsanity.com/paper/2103.04922/) (2021)
- [E(n) Equivariant Normalizing Flows](https://mlsanity.com/paper/2105.09016/) (2021)
- [SE(3)-Stochastic Flow Matching for Protein Backbone Generation](https://mlsanity.com/paper/2310.02391/) (2023)
- [Equivariant flow matching](https://mlsanity.com/paper/2306.15030/) (2023)
- [GENIE: Higher-Order Denoising Diffusion Solvers](https://mlsanity.com/paper/2210.05475/) (2022)
- [An Introduction to Deep Generative Modeling](https://mlsanity.com/paper/2103.05180/) (2021)
- [Neural Flows: Efficient Alternative to Neural ODEs](https://mlsanity.com/paper/2110.13040/) (2021)
- [Action Matching: Learning Stochastic Dynamics from Samples](https://mlsanity.com/paper/2210.06662/) (2022)
- [PeRFlow: Piecewise Rectified Flow as Universal Plug-and-Play Accelerator](https://mlsanity.com/paper/2405.07510/) (2024)
- [Minimizing Trajectory Curvature of ODE-based Generative Models](https://mlsanity.com/paper/2301.12003/) (2023)
- [JFB: Jacobian-Free Backpropagation for Implicit Networks](https://mlsanity.com/paper/2103.12803/) (2021)
- [STaSy: Score-based Tabular data Synthesis](https://mlsanity.com/paper/2210.04018/) (2022)
- [Beltrami Flow and Neural Diffusion on Graphs](https://mlsanity.com/paper/2110.09443/) (2021)
- [Simulation-free Schr"odinger bridges via score and flow matching](https://mlsanity.com/paper/2307.03672/) (2023)
- [Maximum Likelihood Training for Score-Based Diffusion ODEs by High-Order Denoising Score Matching](https://mlsanity.com/paper/2206.08265/) (2022)
- [CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesis](https://mlsanity.com/paper/2304.12654/) (2023)
- [Metric Flow Matching for Smooth Interpolations on the Data Manifold](https://mlsanity.com/paper/2405.14780/) (2024)
- [Stabilizing Equilibrium Models by Jacobian Regularization](https://mlsanity.com/paper/2106.14342/) (2021)
- [Neural Conservation Laws: A Divergence-Free Perspective](https://mlsanity.com/paper/2210.01741/) (2022)
- [Recent Advances in Optimal Transport for Machine Learning](https://mlsanity.com/paper/2306.16156/) (2023)
- [Closed-form Continuous-time Neural Models](https://mlsanity.com/paper/2106.13898/) (2021)
- [Manifold Interpolating Optimal-Transport Flows for Trajectory Inference](https://mlsanity.com/paper/2206.14928/) (2022)
- [Improved Techniques for Maximum Likelihood Estimation for Diffusion ODEs](https://mlsanity.com/paper/2305.03935/) (2023)
- [Sequential Neural Score Estimation: Likelihood-Free Inference with Conditional Score Based Diffusion Models](https://mlsanity.com/paper/2210.04872/) (2022)

## Abstract (arXiv)

Training neural ODEs on large datasets has not been tractable due to the necessity of allowing the adaptive numerical ODE solver to refine its step size to very small values. In practice this leads to dynamics equivalent to many hundreds or even thousands of layers. In this paper, we overcome this apparent difficulty by introducing a theoretically-grounded combination of both optimal transport and stability regularizations which encourage neural ODEs to prefer simpler dynamics out of all the dynamics that solve a problem well. Simpler dynamics lead to faster convergence and to fewer discretizations of the solver, considerably decreasing wall-clock time without loss in performance. Our approach allows us to train neural ODE-based generative models to the same performance as the unregularized dynamics, with significant reductions in training time. This brings neural ODEs closer to practical relevance in large-scale applications.