mlsanityData to 14 Oct 2026

In-Context Learning of Stochastic Differential Equations with Foundation Inference Models

arXiv 2502.19049 ↗First posted 26 Feb 2025cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Stochastic differential equations (SDEs) describe dynamical systems where deterministic flows, governed by a drift function, are superimposed with random fluctuations, dictated by a diffusion function. The accurate estimation (or discovery) of these functions from data is a central problem in machine learning, with wide application across the natural and social sciences. Yet current solutions either rely heavily on prior knowledge of the dynamics or involve intricate training procedures. We introduce FIM-SDE (Foundation Inference Model for SDEs), a pretrained recognition model that delivers accurate in-context (or zero-shot) estimation of the drift and diffusion functions of low-dimensional SDEs, from noisy time series data, and allows rapid finetuning to target datasets. Leveraging concepts from amortized inference and neural operators, we (pre)train FIM-SDE in a supervised fashion to map a large set of noisy, discretely observed SDE paths onto the space of drift and diffusion functions. We demonstrate that FIM-SDE achieves robust in-context function estimation across a wide range of synthetic and real-world processes -- from canonical SDE systems (e.g., double-well dynamics or weakly perturbed Lorenz attractors) to stock price recordings and oil-price and wind-speed fluctuations -- while matching the performance of symbolic, Gaussian process and Neural SDE baselines trained on the target datasets. When finetuned to the target processes, we show that FIM-SDE consistently outperforms all these baselines.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 38 of its 65 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

38 of 65 references.

References · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  4. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  5. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  6. 2018Conditional Neural Processescited by 426
  7. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 402
  8. 2018Efficient Attention: Attention with Linear Complexitiescited by 368
  9. 2021Transformers Can Do Bayesian Inferencecited by 360
  10. 2018Neural Processescited by 311
  11. 2019Attentive Neural Processescited by 283
  12. 2020Scalable Gradients for Stochastic Differential Equationscited by 271
26 more
  1. 2019Deep Learning for Symbolic Mathematicscited by 270
  2. 2021Neural SDEs as Infinite-Dimensional GANscited by 161
  3. 2016Towards a Neural Statisticiancited by 121
  4. 2023ForecastPFN: Synthetically-Trained Zero-Shot Forecastingcited by 104
  5. 2019Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noisecited by 101
  6. 2019Theoretical guarantees for sampling and inference in generative models with latent diffusionscited by 97
  7. 2016Kernels for sequentially ordered datacited by 84
  8. 2017Sparse learning of stochastic dynamic equationscited by 76
  9. 2021Efficient and Accurate Gradients for Neural SDEscited by 71
  10. 2023ODEFormer: Symbolic Regression of Dynamical Systems with Transformerscited by 62
  11. 2022Modeling Temporal Data as Continuous Functions with Stochastic Process Diffusioncited by 39
  12. 2019Learning interpretable continuous-time models of latent stochastic dynamical systemscited by 30
  13. 2018The Variational Homoencoder: Learning to learn high capacity generative models from few examplescited by 23
  14. 2025SDE Matching: Scalable and Simulation-Free Training of Latent Stochastic Differential Equationscited by 23
  15. 2016Inference Networks for Sequential Monte Carlo in Graphical Modelscited by 20
  16. 2021Dual Parameterization of Sparse Variational Gaussian Processescited by 19
  17. 2023Latent SDEs on Homogeneous Spacescited by 16
  18. 2025Amortized In-Context Bayesian Posterior Estimationcited by 15
  19. 2024Foundation Inference Models for Markov Jump Processescited by 13
  20. 2017Non-parametric Estimation of Stochastic Differential Equations with Sparse Gaussian Processescited by 12
  21. 2017Approximate Bayes learning of stochastic differential equationscited by 11
  22. 2023Variational Gaussian Process Diffusion Processescited by 11
  23. 2022The future is different: Large pre-trained language models fail in prediction taskscited by 8
  24. 2023Neural Dynamic Focused Topic Modelcited by 8
  25. 2021Moment-Based Variational Inference for Stochastic Differential Equationscited by 6
  26. 2025In-Context Parametric Inference: Point or Distribution Estimators?cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 4.

7 papers.

  1. 2025In-Context Learning of Temporal Point Processes with Foundation Inference Modelscited by 10
  2. 2026Domain-Adaptation through Synthetic Data: Fine-Tuning Large Language Models for German Lawcited by 2
  3. 2025Towards Fast Coarse-graining and Equation Discovery with Foundation Inference Modelscited by 1
  4. 2025Towards Foundation Inference Models that Learn ODEs In-Contextcited by 0
  5. 2026EVIL: Evolving Interpretable Algorithms for Zero-Shot Inference on Event Sequences and Time Series with LLMscited by 0
  6. 2026Prior-Fitted Functional Flow: In-Context Generative Models for Pharmacokineticscited by 0
  7. 2026Learning Continuous Neural Representation of Stochastic Hybrid Systemscited by 0