# Implementation and (Inverse Modified) Error Analysis for implicitly-templated ODE-nets

arXiv: https://arxiv.org/abs/2303.17824 · HTML: https://mlsanity.com/paper/2303.17824/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 56 references that are in the map.
- First posted on arXiv 31 Mar 2023.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Invertible Residual Networks](https://mlsanity.com/paper/1811.00995/) (2018)
- [Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1708.00588/) (2017)
- [Symplectic Recurrent Neural Networks](https://mlsanity.com/paper/1909.13334/) (2019)
- [Hamiltonian Generative Networks](https://mlsanity.com/paper/1909.13789/) (2019)
- [Implicit Deep Learning](https://mlsanity.com/paper/1908.06315/) (2019)
- [Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems](https://mlsanity.com/paper/1801.01236/) (2018)
- [On Training Implicit Models](https://mlsanity.com/paper/2111.05177/) (2021)
- [GFINNs: GENERIC Formalism Informed Neural Networks for Deterministic and Stochastic Dynamical Systems](https://mlsanity.com/paper/2109.00092/) (2021)
- [Nonparametric inference of interaction laws in systems of agents from trajectory data](https://mlsanity.com/paper/1812.06003/) (2018)
- [Data-driven Prediction of General Hamiltonian Dynamics via Learning Exactly-Symplectic Maps](https://mlsanity.com/paper/2103.05632/) (2021)
- [OnsagerNet: Learning Stable and Interpretable Dynamics using a Generalized Onsager Principle](https://mlsanity.com/paper/2009.02327/) (2020)
- [Hypersolvers: Toward Fast Continuous-Depth Models](https://mlsanity.com/paper/2007.09601/) (2020)
- [Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics](https://mlsanity.com/paper/2105.03918/) (2021)
- [Which priors matter? Benchmarking models for learning latent dynamics](https://mlsanity.com/paper/2111.05458/) (2021)
- [Numerical Aspects for Approximating Governing Equations Using Data](https://mlsanity.com/paper/1809.09170/) (2018)
- [On Numerical Integration in Neural Ordinary Differential Equations](https://mlsanity.com/paper/2206.07335/) (2022)
- [Time-Reversal Symmetric ODE Network](https://mlsanity.com/paper/2007.11362/) (2020)
- [Structure-preserving Method for Reconstructing Unknown Hamiltonian Systems from Trajectory Data](https://mlsanity.com/paper/1905.10396/) (2019)
- [Discovery of Dynamics Using Linear Multistep Methods](https://mlsanity.com/paper/1912.12728/) (2019)
- [Revealing hidden dynamics from time-series data by ODENet](https://mlsanity.com/paper/2005.04849/) (2020)
- [The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation](https://mlsanity.com/paper/2103.11488/) (2021)
- [Partial observations and conservation laws: Grey-box modeling in biotechnology and optogenetics](https://mlsanity.com/paper/1909.04234/) (2019)
- [Personalized Algorithm Generation: A Case Study in Learning ODE Integrators](https://mlsanity.com/paper/2105.01303/) (2021)
- [A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms](https://mlsanity.com/paper/2211.12386/) (2022)

## Built on by (most cited first, 6 of 6)

- [Correcting model misspecification in physics-informed neural networks (PINNs)](https://mlsanity.com/paper/2310.10776/) (2023)
- [AI-Lorenz: A physics-data-driven framework for black-box and gray-box identification of chaotic systems with symbolic regression](https://mlsanity.com/paper/2312.14237/) (2023)
- [DUE: A Deep Learning Framework and Library for Modeling Unknown Equations](https://mlsanity.com/paper/2504.10373/) (2025)
- [Some of the variables, some of the parameters, some of the times, with some physics known: Identification with partial information](https://mlsanity.com/paper/2304.14214/) (2023)
- [A Mechanistic Analysis of Transformers for Dynamical Systems](https://mlsanity.com/paper/2512.21113/) (2025)
- [Learning symplectic model reduction based on an approximation theorem of symplectic embeddings](https://mlsanity.com/paper/2606.04623/) (2026)

## Abstract (arXiv)

We focus on learning unknown dynamics from data using ODE-nets templated on implicit numerical initial value problem solvers. First, we perform Inverse Modified error analysis of the ODE-nets using unrolled implicit schemes for ease of interpretation. It is shown that training an ODE-net using an unrolled implicit scheme returns a close approximation of an Inverse Modified Differential Equation (IMDE). In addition, we establish a theoretical basis for hyper-parameter selection when training such ODE-nets, whereas current strategies usually treat numerical integration of ODE-nets as a black box. We thus formulate an adaptive algorithm which monitors the level of error and adapts the number of (unrolled) implicit solution iterations during the training process, so that the error of the unrolled approximation is less than the current learning loss. This helps accelerate training, while maintaining accuracy. Several numerical experiments are performed to demonstrate the advantages of the proposed algorithm compared to nonadaptive unrollings, and validate the theoretical analysis. We also note that this approach naturally allows for incorporating partially known physical terms in the equations, giving rise to what is termed ``gray box" identification.