# Differentiable Multiple Shooting Layers

arXiv: https://arxiv.org/abs/2106.03885 · HTML: https://mlsanity.com/paper/2106.03885/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 51 references that are in the map.
- First posted on arXiv 7 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [Scalable Gradients for Stochastic Differential Equations](https://mlsanity.com/paper/2001.01328/) (2020)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [Neural Jump Stochastic Differential Equations](https://mlsanity.com/paper/1905.10403/) (2019)
- [PPINN: Parareal Physics-Informed Neural Network for time-dependent PDEs](https://mlsanity.com/paper/1909.10145/) (2019)
- [Layer-Parallel Training of Deep Residual Neural Networks](https://mlsanity.com/paper/1812.04352/) (2018)
- [DiffEqFlux.jl - A Julia Library for Neural Differential Equations](https://mlsanity.com/paper/1902.02376/) (2019)
- [Hypersolvers: Toward Fast Continuous-Depth Models](https://mlsanity.com/paper/2007.09601/) (2020)
- [Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics](https://mlsanity.com/paper/2105.03918/) (2021)
- [TorchDyn: A Neural Differential Equations Library](https://mlsanity.com/paper/2009.09346/) (2020)
- [A Shooting Formulation of Deep Learning](https://mlsanity.com/paper/2006.10330/) (2020)
- [Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigrid](https://mlsanity.com/paper/2007.07336/) (2020)
- [Multiple-shooting adjoint method for whole-brain dynamic causal modeling](https://mlsanity.com/paper/2102.11013/) (2021)
- [A Practical Layer-Parallel Training Algorithm for Residual Networks](https://mlsanity.com/paper/2009.01462/) (2020)

## Built on by (most cited first, 20 of 20)

- [Monarch: Expressive Structured Matrices for Efficient and Accurate Training](https://mlsanity.com/paper/2204.00595/) (2022)
- [Effectively Modeling Time Series with Simple Discrete State Spaces](https://mlsanity.com/paper/2303.09489/) (2023)
- [Parallelizing non-linear sequential models over the sequence length](https://mlsanity.com/paper/2309.12252/) (2023)
- [Deep Latent State Space Models for Time-Series Generation](https://mlsanity.com/paper/2212.12749/) (2022)
- [Towards Scalable and Stable Parallelization of Nonlinear RNNs](https://mlsanity.com/paper/2407.19115/) (2024)
- [Physics-Informed Machine Learning for Modeling and Control of Dynamical Systems](https://mlsanity.com/paper/2306.13867/) (2023)
- [State-Free Inference of State-Space Models: The Transfer Function Approach](https://mlsanity.com/paper/2405.06147/) (2024)
- [Latent Neural ODEs with Sparse Bayesian Multiple Shooting](https://mlsanity.com/paper/2210.03466/) (2022)
- [Variational multiple shooting for Bayesian ODEs with Gaussian processes](https://mlsanity.com/paper/2106.10905/) (2021)
- [Homotopy-based training of NeuralODEs for accurate dynamics discovery](https://mlsanity.com/paper/2210.01407/) (2022)
- [DeepPCR: Parallelizing Sequential Operations in Neural Networks](https://mlsanity.com/paper/2309.16318/) (2023)
- [Learning Physics Informed Neural ODEs With Partial Measurements](https://mlsanity.com/paper/2412.08681/) (2024)
- [Self-Similarity Priors: Neural Collages as Differentiable Fractal Representations](https://mlsanity.com/paper/2204.07673/) (2022)
- [Neural Solvers for Fast and Accurate Numerical Optimal Control](https://mlsanity.com/paper/2203.08072/) (2022)
- [Hybridizing Physics and Neural ODEs for Predicting Plasma Inductance Dynamics in Tokamak Fusion Reactors](https://mlsanity.com/paper/2310.20079/) (2023)
- [Trajectory Stitching for Solving Inverse Problems with Flow-Based Models](https://mlsanity.com/paper/2602.08538/) (2026)
- [Learning Latent Dynamics via Invariant Decomposition and (Spatio-)Temporal Transformers](https://mlsanity.com/paper/2306.12077/) (2023)
- [A condensing approach to multiple shooting neural ordinary differential equation](https://mlsanity.com/paper/2506.00724/) (2025)
- [Long-Time Trajectory Approximation via SA-NODEs: Model Predictive and Floquet Strategies](https://mlsanity.com/paper/2608.10738/) (2026)
- [Neural ODEs Meet Concurrent Learning: Stable Online Learning with Lyapunov Guarantees](https://mlsanity.com/paper/2609.32289/) (2026)

## Abstract (arXiv)

We detail a novel class of implicit neural models. Leveraging time-parallel methods for differential equations, Multiple Shooting Layers (MSLs) seek solutions of initial value problems via parallelizable root-finding algorithms. MSLs broadly serve as drop-in replacements for neural ordinary differential equations (Neural ODEs) with improved efficiency in number of function evaluations (NFEs) and wall-clock inference time. We develop the algorithmic framework of MSLs, analyzing the different choices of solution methods from a theoretical and computational perspective. MSLs are showcased in long horizon optimal control of ODEs and PDEs and as latent models for sequence generation. Finally, we investigate the speedups obtained through application of MSL inference in neural controlled differential equations (Neural CDEs) for time series classification of medical data.