# Closed-form Continuous-time Neural Models

arXiv: https://arxiv.org/abs/2106.13898 · HTML: https://mlsanity.com/paper/2106.13898/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 88 references that are in the map.
- First posted on arXiv 25 Jun 2021.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models](https://mlsanity.com/paper/1810.01367/) (2018)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Learning the solution operator of parametric partial differential equations with physics-informed DeepOnets](https://mlsanity.com/paper/2103.10974/) (2021)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attention](https://mlsanity.com/paper/2102.03902/) (2021)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Scalable Gradients for Stochastic Differential Equations](https://mlsanity.com/paper/2001.01328/) (2020)
- [Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks](https://mlsanity.com/paper/1701.05923/) (2017)
- [Multi-Time Attention Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2101.10318/) (2021)
- [How to train your neural ODE: the world of Jacobian and kinetic regularization](https://mlsanity.com/paper/2002.02798/) (2020)
- [Latent ODEs for Irregularly-Sampled Time Series](https://mlsanity.com/paper/1907.03907/) (2019)
- [Liquid Time-constant Networks](https://mlsanity.com/paper/2006.04439/) (2020)
- [Stochastic Normalizing Flows](https://mlsanity.com/paper/2002.06707/) (2020)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [Riemannian Continuous Normalizing Flows](https://mlsanity.com/paper/2006.10605/) (2020)
- [Set Functions for Time Series](https://mlsanity.com/paper/1909.12064/) (2019)
- [Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group](https://mlsanity.com/paper/1901.08428/) (2019)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences](https://mlsanity.com/paper/1610.09513/) (2016)
- [Interpolation-Prediction Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/1909.07782/) (2019)
- [Lipschitz Recurrent Neural Networks](https://mlsanity.com/paper/2006.12070/) (2020)

## Built on by (most cited first, 30 of 65)

- [Liquid Structural State-Space Models](https://mlsanity.com/paper/2209.12951/) (2022)
- [Dataset Distillation with Convexified Implicit Gradients](https://mlsanity.com/paper/2302.06755/) (2023)
- [Hierarchical State Space Models for Continuous Sequence-to-Sequence Modeling](https://mlsanity.com/paper/2402.10211/) (2024)
- [On the Forward Invariance of Neural ODEs](https://mlsanity.com/paper/2210.04763/) (2022)
- [NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?](https://mlsanity.com/paper/2606.24530/) (2026)
- [Kelp: A Streaming Safeguard for Large Models via Latent Dynamics-Guided Risk Detection](https://mlsanity.com/paper/2510.09694/) (2025)
- [EBES: Easy Benchmarking for Event Sequences](https://mlsanity.com/paper/2410.03399/) (2024)
- [Gated Recurrent Neural Networks with Weighted Time-Delay Feedback](https://mlsanity.com/paper/2212.00228/) (2022)
- [Recovering implicit physics model under real-world constraints](https://mlsanity.com/paper/2412.02215/) (2024)
- [Beyond representational alignment with brain-guided language models for robust reasoning](https://mlsanity.com/paper/2606.11893/) (2026)
- [SLS4D: Sparse Latent Space for 4D Novel View Synthesis](https://mlsanity.com/paper/2312.09743/) (2023)
- [Liquid Fourier Latent Dynamics Networks for fast GPU-based numerical simulations in computational cardiology](https://mlsanity.com/paper/2408.09818/) (2024)
- [Accuracy, Memory Efficiency and Generalization: A Comparative Study on Liquid Neural Networks and Recurrent Neural Networks](https://mlsanity.com/paper/2510.07578/) (2025)
- [Neuronal Attention Circuit (NAC) for Representation Learning](https://mlsanity.com/paper/2512.10282/) (2025)
- [EntroLnn: Entropy-Guided Liquid Neural Networks for Operando Refinement of Battery Capacity Fade Trajectories](https://mlsanity.com/paper/2601.06195/) (2026)
- [Symbolic-Vector Attention Fusion for Collective Intelligence](https://mlsanity.com/paper/2604.03955/) (2026)
- [Learning Continuous Network Emerging Dynamics from Scarce Observations via Data-Adaptive Stochastic Processes](https://mlsanity.com/paper/2310.16466/) (2023)
- [Physics-Informed Calibration of Aeromagnetic Compensation in Magnetic Navigation Systems using Liquid Time-Constant Networks](https://mlsanity.com/paper/2401.09631/) (2024)
- [Deep learning for predicting the occurrence of tipping points](https://mlsanity.com/paper/2407.18693/) (2024)
- [Building Trustworthy AI for Materials Discovery: From Autonomous Laboratories to Z-scores](https://mlsanity.com/paper/2512.01080/) (2025)
- [Deep Sequence Modeling with Quantum Dynamics: Language as a Wave Function](https://mlsanity.com/paper/2602.22255/) (2026)
- [Systematic construction of continuous-time neural networks for linear dynamical systems](https://mlsanity.com/paper/2403.16215/) (2024)
- [A Neural Network Training Method Based on Neuron Connection Coefficient Adjustments](https://mlsanity.com/paper/2502.10414/) (2025)
- [Learning Spatio-Temporal Dynamics for Trajectory Recovery via Time-Aware Transformer](https://mlsanity.com/paper/2505.13857/) (2025)
- [Riemannian Liquid Spatio-Temporal Graph Network](https://mlsanity.com/paper/2601.14115/) (2026)
- [Oscillators Are All You Need: Irregular Time Series Modelling via Damped Harmonic Oscillators with Closed-Form Solutions](https://mlsanity.com/paper/2602.12139/) (2026)
- [MeloTune: On-Device Arousal Learning and Peer-to-Peer Mood Coupling for Proactive Music Curation](https://mlsanity.com/paper/2604.10815/) (2026)
- [LiquidTAD: Efficient Temporal Action Detection via Parallel Liquid-Inspired Temporal Relaxation](https://mlsanity.com/paper/2604.18274/) (2026)
- [Mesh Memory Protocol: Semantic Infrastructure for Multi-Agent LLM Systems](https://mlsanity.com/paper/2604.19540/) (2026)
- [FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learning](https://mlsanity.com/paper/2605.04421/) (2026)

## Abstract (arXiv)

Continuous-time neural processes are performant sequential decision-makers that are built by differential equations (DE). However, their expressive power when they are deployed on computers is bottlenecked by numerical DE solvers. This limitation has significantly slowed down the scaling and understanding of numerous natural physical phenomena such as the dynamics of nervous systems. Ideally, we would circumvent this bottleneck by solving the given dynamical system in closed form. This is known to be intractable in general. Here, we show it is possible to closely approximate the interaction between neurons and synapses -- the building blocks of natural and artificial neural networks -- constructed by liquid time-constant networks (LTCs) efficiently in closed-form. To this end, we compute a tightly-bounded approximation of the solution of an integral appearing in LTCs' dynamics, that has had no known closed-form solution so far. This closed-form solution substantially impacts the design of continuous-time and continuous-depth neural models; for instance, since time appears explicitly in closed-form, the formulation relaxes the need for complex numerical solvers. Consequently, we obtain models that are between one and five orders of magnitude faster in training and inference compared to differential equation-based counterparts. More importantly, in contrast to ODE-based continuous networks, closed-form networks can scale remarkably well compared to other deep learning instances. Lastly, as these models are derived from liquid networks, they show remarkable performance in time series modeling, compared to advanced recurrent models.