# Zero-shot Imputation with Foundation Inference Models for Dynamical Systems

arXiv: https://arxiv.org/abs/2402.07594 · HTML: https://mlsanity.com/paper/2402.07594/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 67 references that are in the map.
- First posted on arXiv 12 Feb 2024.

## Builds on (most cited first, 30 of 33)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation](https://mlsanity.com/paper/2107.03502/) (2021)
- [Recurrent Neural Networks for Multivariate Time Series with Missing Values](https://mlsanity.com/paper/1606.01865/) (2016)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Data Distributional Properties Drive Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2205.05055/) (2022)
- [BRITS: Bidirectional Recurrent Imputation for Time Series](https://mlsanity.com/paper/1805.10572/) (2018)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [Multi-Time Attention Networks for Irregularly Sampled Time Series](https://mlsanity.com/paper/2101.10318/) (2021)
- [Contemporary Symbolic Regression Methods and their Relative Performance](https://mlsanity.com/paper/2107.14351/) (2021)
- [SAITS: Self-Attention-based Imputation for Time Series](https://mlsanity.com/paper/2202.08516/) (2022)
- [Towards a Neural Statistician](https://mlsanity.com/paper/1606.02185/) (2016)
- [Interpretable Scientific Discovery with Symbolic Regression: A Review](https://mlsanity.com/paper/2211.10873/) (2022)
- [The Capacity and Robustness Trade-off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2304.05206/) (2023)
- [A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement](https://mlsanity.com/paper/2204.09308/) (2022)
- [ODEFormer: Symbolic Regression of Dynamical Systems with Transformers](https://mlsanity.com/paper/2310.05573/) (2023)
- [Neural ODE Processes](https://mlsanity.com/paper/2103.12413/) (2021)
- [Deep Learning for Multivariate Time Series Imputation: A Survey](https://mlsanity.com/paper/2402.04059/) (2024)
- [Sequential Neural Processes](https://mlsanity.com/paper/1906.10264/) (2019)
- [Learning unknown ODE models with Gaussian processes](https://mlsanity.com/paper/1803.04303/) (2018)
- [Meta-Learning Dynamics Forecasting Using Task Inference](https://mlsanity.com/paper/2102.10271/) (2021)
- [Predicting Ordinary Differential Equations with Transformers](https://mlsanity.com/paper/2307.12617/) (2023)
- [The Variational Homoencoder: Learning to learn high capacity generative models from few examples](https://mlsanity.com/paper/1807.08919/) (2018)

## Built on by (most cited first, 13 of 13)

- [SSD-TS: Exploring the Potential of Linear State Space Models for Diffusion Models in Time Series Imputation](https://mlsanity.com/paper/2410.13338/) (2024)
- [Panda: A pretrained forecast model for chaotic dynamics](https://mlsanity.com/paper/2505.13755/) (2025)
- [In-Context Learning of Temporal Point Processes with Foundation Inference Models](https://mlsanity.com/paper/2509.24762/) (2025)
- [On Foundation Models for Dynamical Systems from Purely Synthetic Data](https://mlsanity.com/paper/2412.00395/) (2024)
- [Amortized In-Context Mixed Effect Transformer Models: A Zero-Shot Approach for Pharmacokinetics](https://mlsanity.com/paper/2508.15659/) (2025)
- [Position: Biology is the Challenge Physics-Informed ML Needs to Evolve](https://mlsanity.com/paper/2510.25368/) (2025)
- [Foundation Inference Models for Ordinary Differential Equations](https://mlsanity.com/paper/2602.08733/) (2026)
- [Learning System Dynamics without Forgetting](https://mlsanity.com/paper/2407.00717/) (2024)
- [On Foundation Models for Temporal Point Processes to Accelerate Scientific Discovery](https://mlsanity.com/paper/2510.12640/) (2025)
- [Towards Foundation Inference Models that Learn ODEs In-Context](https://mlsanity.com/paper/2510.12650/) (2025)
- [CINDI: Conditional Imputation and Noisy Data Integrity with Flows in Power Grid Data](https://mlsanity.com/paper/2603.11745/) (2026)
- [EVIL: Evolving Interpretable Algorithms for Zero-Shot Inference on Event Sequences and Time Series with LLMs](https://mlsanity.com/paper/2604.15787/) (2026)
- [Prior-Fitted Functional Flow: In-Context Generative Models for Pharmacokinetics](https://mlsanity.com/paper/2604.17670/) (2026)

## Abstract (arXiv)

Dynamical systems governed by ordinary differential equations (ODEs) serve as models for a vast number of natural and social phenomena. In this work, we offer a fresh perspective on the classical problem of imputing missing time series data, whose underlying dynamics are assumed to be determined by ODEs. Specifically, we revisit ideas from amortized inference and neural operators, and propose a novel supervised learning framework for zero-shot time series imputation, through parametric functions satisfying some (hidden) ODEs. Our proposal consists of two components. First, a broad probability distribution over the space of ODE solutions, observation times and noise mechanisms, with which we generate a large, synthetic dataset of (hidden) ODE solutions, along with their noisy and sparse observations. Second, a neural recognition model that is trained offline, to map the generated time series onto the spaces of initial conditions and time derivatives of the (hidden) ODE solutions, which we then integrate to impute the missing data. We empirically demonstrate that one and the same (pretrained) recognition model can perform zero-shot imputation across 63 distinct time series with missing values, each sampled from widely different dynamical systems. Likewise, we demonstrate that it can perform zero-shot imputation of missing high-dimensional data in 10 vastly different settings, spanning human motion, air quality, traffic and electricity studies, as well as Navier-Stokes simulations -- without requiring any fine-tuning. What is more, our proposal often outperforms state-of-the-art methods, which are trained on the target datasets. Our pretrained model, repository and tutorials are available online.