# Discretize-Optimize vs. Optimize-Discretize for Time-Series Regression and Continuous Normalizing Flows

arXiv: https://arxiv.org/abs/2005.13420 · HTML: https://mlsanity.com/paper/2005.13420/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2020.

## Built on by (most cited first, 30 of 37)

- [JFB: Jacobian-Free Backpropagation for Implicit Networks](https://mlsanity.com/paper/2103.12803/) (2021)
- [Learning Physical Models that Can Respect Conservation Laws](https://mlsanity.com/paper/2302.11002/) (2023)
- [Learning the Dynamics of Physical Systems from Sparse Observations with Finite Element Networks](https://mlsanity.com/paper/2203.08852/) (2022)
- [Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics](https://mlsanity.com/paper/2105.03918/) (2021)
- [Multiple shooting for training neural differential equations on time series](https://mlsanity.com/paper/2109.06786/) (2021)
- [Taming Hyperparameter Tuning in Continuous Normalizing Flows Using the JKO Scheme](https://mlsanity.com/paper/2211.16757/) (2022)
- [Sensitivity-Constrained Fourier Neural Operators for Forward and Inverse Problems in Parametric Differential Equations](https://mlsanity.com/paper/2505.08740/) (2025)
- [Training Stiff Neural Ordinary Differential Equations with Implicit Single-Step Methods](https://mlsanity.com/paper/2410.05592/) (2024)
- [COMET Flows: Towards Generative Modeling of Multivariate Extremes and Tail Dependence](https://mlsanity.com/paper/2205.01224/) (2022)
- [Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency](https://mlsanity.com/paper/2505.13499/) (2025)
- [Differential Equations for Continuous-Time Deep Learning](https://mlsanity.com/paper/2401.03965/) (2024)
- [Sequential-in-time training of nonlinear parametrizations for solving time-dependent partial differential equations](https://mlsanity.com/paper/2404.01145/) (2024)
- [OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization](https://mlsanity.com/paper/2501.18793/) (2025)
- [Comparison of neural closure models for discretised PDEs](https://mlsanity.com/paper/2210.14675/) (2022)
- [A scalable deep learning approach for solving high-dimensional dynamic optimal transport](https://mlsanity.com/paper/2205.07521/) (2022)
- [Training Stiff Neural Ordinary Differential Equations with Explicit Exponential Integration Methods](https://mlsanity.com/paper/2412.01181/) (2024)
- [Locally Regularized Neural Differential Equations: Some Black Boxes Were Meant to Remain Closed!](https://mlsanity.com/paper/2303.02262/) (2023)
- [Reparameterization Flow Policy Optimization](https://mlsanity.com/paper/2602.03501/) (2026)
- [A memory-efficient neural ODE framework based on high-level adjoint differentiation](https://mlsanity.com/paper/2206.01298/) (2022)
- [Correcting Auto-Differentiation in Neural-ODE Training](https://mlsanity.com/paper/2306.02192/) (2023)
- [System-Aware Neural ODE Processes for Few-Shot Bayesian Optimization](https://mlsanity.com/paper/2406.02352/) (2024)
- [Scientific Machine Learning of Flow Resistance Using Universal Shallow Water Equations with Differentiable Programming](https://mlsanity.com/paper/2502.12396/) (2025)
- [The Vanishing Gradient Problem for Stiff Neural Differential Equations](https://mlsanity.com/paper/2508.01519/) (2025)
- [Trainability, Expressivity and Interpretability in Gated Neural ODEs](https://mlsanity.com/paper/2307.06398/) (2023)
- [Greed is Good: A Unifying Perspective on Guided Generation](https://mlsanity.com/paper/2502.08006/) (2025)
- [Neural Field Thermal Tomography: A Differentiable Physics Framework for Non-Destructive Evaluation](https://mlsanity.com/paper/2603.11045/) (2026)
- [Meta-Solver for Neural Ordinary Differential Equations](https://mlsanity.com/paper/2103.08561/) (2021)
- [KPFlow: An Operator Perspective on Dynamic Collapse Under Gradient Descent Training of Recurrent Networks](https://mlsanity.com/paper/2507.06381/) (2025)
- [Neural Discovery of Memory and Nonlocal Kernels in Integro-Differential Equations with Constrained Kolmogorov--Arnold Networks](https://mlsanity.com/paper/2607.11110/) (2026)
- [Weight-Parameterization in Continuous Time Deep Neural Networks for Surrogate Modeling](https://mlsanity.com/paper/2507.22045/) (2025)

## Abstract (arXiv)

We compare the discretize-optimize (Disc-Opt) and optimize-discretize (Opt-Disc) approaches for time-series regression and continuous normalizing flows (CNFs) using neural ODEs. Neural ODEs are ordinary differential equations (ODEs) with neural network components. Training a neural ODE is an optimal control problem where the weights are the controls and the hidden features are the states. Every training iteration involves solving an ODE forward and another backward in time, which can require large amounts of computation, time, and memory. Comparing the Opt-Disc and Disc-Opt approaches in image classification tasks, Gholami et al. (2019) suggest that Disc-Opt is preferable due to the guaranteed accuracy of gradients. In this paper, we extend the comparison to neural ODEs for time-series regression and CNFs. Unlike in classification, meaningful models in these tasks must also satisfy additional requirements beyond accurate final-time output, e.g., the invertibility of the CNF. Through our numerical experiments, we demonstrate that with careful numerical treatment, Disc-Opt methods can achieve similar performance as Opt-Disc at inference with drastically reduced training costs. Disc-Opt reduced costs in six out of seven separate problems with training time reduction ranging from 39% to 97%, and in one case, Disc-Opt reduced training from nine days to less than one day.