# An Ode to an ODE

arXiv: https://arxiv.org/abs/2006.11421 · HTML: https://mlsanity.com/paper/2006.11421/

## Key facts

- Cited by 17 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 19 Jun 2020.

## Built on by (most cited first, 17 of 17)

- [Hamiltonian Deep Neural Networks Guaranteeing Non-vanishing Gradients by Design](https://mlsanity.com/paper/2105.13205/) (2021)
- [Self-Consistency of the Fokker-Planck Equation](https://mlsanity.com/paper/2206.00860/) (2022)
- [Variational multiple shooting for Bayesian ODEs with Gaussian processes](https://mlsanity.com/paper/2106.10905/) (2021)
- [Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules](https://mlsanity.com/paper/2206.01649/) (2022)
- [Homotopy-based training of NeuralODEs for accurate dynamics discovery](https://mlsanity.com/paper/2210.01407/) (2022)
- [Robust Classification using Contractive Hamiltonian Neural ODEs](https://mlsanity.com/paper/2203.11805/) (2022)
- [Distributed neural network control with dependability guarantees: a compositional port-Hamiltonian approach](https://mlsanity.com/paper/2112.09046/) (2021)
- [Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning](https://mlsanity.com/paper/2503.01329/) (2025)
- [RNNs of RNNs: Recursive Construction of Stable Assemblies of Recurrent Neural Networks](https://mlsanity.com/paper/2106.08928/) (2021)
- [Learning Energy Conserving Dynamics Efficiently with Hamiltonian Gaussian Processes](https://mlsanity.com/paper/2303.01925/) (2023)
- [Uncertainty and Structure in Neural Ordinary Differential Equations](https://mlsanity.com/paper/2305.13290/) (2023)
- [Meta-Reinforcement Learning with Self-Modifying Networks](https://mlsanity.com/paper/2202.02363/) (2022)
- [Foundation Inference Models for Ordinary Differential Equations](https://mlsanity.com/paper/2602.08733/) (2026)
- [Learning ODEs via Diffeomorphisms for Fast and Robust Integration](https://mlsanity.com/paper/2107.01650/) (2021)
- [The Disappearance of Timestep Embedding in Modern Time-Dependent Neural Networks](https://mlsanity.com/paper/2405.14126/) (2024)
- [Deep Neural Networks Inspired by Differential Equations](https://mlsanity.com/paper/2510.09685/) (2025)
- [A Constructive Approach to Function Realization by Neural Stochastic Differential Equations](https://mlsanity.com/paper/2307.00215/) (2023)

## Abstract (arXiv)

We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the orthogonal group O(d). This nested system of two flows, where the parameter-flow is constrained to lie on the compact manifold, provides stability and effectiveness of training and provably solves the gradient vanishing-explosion problem which is intrinsically related to training deep neural network architectures such as Neural ODEs. Consequently, it leads to better downstream models, as we show on the example of training reinforcement learning policies with evolution strategies, and in the supervised learning setting, by comparing with previous SOTA baselines. We provide strong convergence results for our proposed mechanism that are independent of the depth of the network, supporting our empirical studies. Our results show an intriguing connection between the theory of deep neural networks and the field of matrix flows on compact manifolds.