# Hypersolvers: Toward Fast Continuous-Depth Models

arXiv: https://arxiv.org/abs/2007.09601 · HTML: https://mlsanity.com/paper/2007.09601/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Jul 2020.

## Built on by (most cited first, 30 of 33)

- [Monarch: Expressive Structured Matrices for Efficient and Accurate Training](https://mlsanity.com/paper/2204.00595/) (2022)
- [Neural Flows: Efficient Alternative to Neural ODEs](https://mlsanity.com/paper/2110.13040/) (2021)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [Stabilizing Equilibrium Models by Jacobian Regularization](https://mlsanity.com/paper/2106.14342/) (2021)
- [Closed-form Continuous-time Neural Models](https://mlsanity.com/paper/2106.13898/) (2021)
- [SEGNO: Generalizing Equivariant Graph Neural Networks with Physical Inductive Biases](https://mlsanity.com/paper/2308.13212/) (2023)
- [Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics](https://mlsanity.com/paper/2105.03918/) (2021)
- [Neural Integral Equations](https://mlsanity.com/paper/2209.15190/) (2022)
- [On Numerical Integration in Neural Ordinary Differential Equations](https://mlsanity.com/paper/2206.07335/) (2022)
- [Differentiable Multiple Shooting Layers](https://mlsanity.com/paper/2106.03885/) (2021)
- [Transform Once: Efficient Operator Learning in Frequency Domain](https://mlsanity.com/paper/2211.14453/) (2022)
- [Sparse Flows: Pruning Continuous-depth Models](https://mlsanity.com/paper/2106.12718/) (2021)
- [Taylor-Lagrange Neural Ordinary Differential Equations: Toward Fast Training and Evaluation of Neural ODEs](https://mlsanity.com/paper/2201.05715/) (2022)
- [Convergent Graph Solvers](https://mlsanity.com/paper/2106.01680/) (2021)
- [GoTube: Scalable Stochastic Verification of Continuous-Depth Models](https://mlsanity.com/paper/2107.08467/) (2021)
- [Unsupervised Reservoir Computing for Solving Ordinary Differential Equations](https://mlsanity.com/paper/2108.11417/) (2021)
- [On the Forward Invariance of Neural ODEs](https://mlsanity.com/paper/2210.04763/) (2022)
- [Neural Fixed-Point Acceleration for Convex Optimization](https://mlsanity.com/paper/2107.10254/) (2021)
- [PARCv2: Physics-aware Recurrent Convolutional Neural Networks for Spatiotemporal Dynamics Modeling](https://mlsanity.com/paper/2402.12503/) (2024)
- [Accelerating Neural ODEs Using Model Order Reduction](https://mlsanity.com/paper/2105.14070/) (2021)
- [Learning context-aware adaptive solvers to accelerate quadratic programming](https://mlsanity.com/paper/2211.12443/) (2022)
- [Continuous Spatiotemporal Transformers](https://mlsanity.com/paper/2301.13338/) (2023)
- [Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs](https://mlsanity.com/paper/2204.08621/) (2022)
- [Implementation and (Inverse Modified) Error Analysis for implicitly-templated ODE-nets](https://mlsanity.com/paper/2303.17824/) (2023)
- [Strong Stochastic Flow Maps](https://mlsanity.com/paper/2606.01086/) (2026)
- [Neural Solvers for Fast and Accurate Numerical Optimal Control](https://mlsanity.com/paper/2203.08072/) (2022)
- [Locally Regularized Neural Differential Equations: Some Black Boxes Were Meant to Remain Closed!](https://mlsanity.com/paper/2303.02262/) (2023)
- [Wassersplines for Neural Vector Field--Controlled Animation](https://mlsanity.com/paper/2201.11940/) (2022)
- [IGNN-Solver: A Graph Neural Solver for Implicit Graph Neural Networks](https://mlsanity.com/paper/2410.08524/) (2024)
- [Adaptive Asynchronous Control Using Meta-learned Neural Ordinary Differential Equations](https://mlsanity.com/paper/2207.12062/) (2022)

## Abstract (arXiv)

The infinite-depth paradigm pioneered by Neural ODEs has launched a renaissance in the search for novel dynamical system-inspired deep learning primitives; however, their utilization in problems of non-trivial size has often proved impossible due to poor computational scalability. This work paves the way for scalable Neural ODEs with time-to-prediction comparable to traditional discrete networks. We introduce hypersolvers, neural networks designed to solve ODEs with low overhead and theoretical guarantees on accuracy. The synergistic combination of hypersolvers and Neural ODEs allows for cheap inference and unlocks a new frontier for practical application of continuous-depth models. Experimental evaluations on standard benchmarks, such as sampling for continuous normalizing flows, reveal consistent pareto efficiency over classical numerical methods.