# In-Context Learning for System Identification: Probabilistic Formulation, Long-Sequence Handling, and Benchmark Validation

arXiv: https://arxiv.org/abs/2610.13411 · HTML: https://mlsanity.com/paper/2610.13411/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [How to train your MAML](https://mlsanity.com/paper/1810.09502/) (2018)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023)
- [Visual In-Context Learning for Large Vision-Language Models](https://mlsanity.com/paper/2402.11574/) (2024)
- [General-Purpose In-Context Learning by Meta-Learning Transformers](https://mlsanity.com/paper/2212.04458/) (2022)
- [SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation](https://mlsanity.com/paper/2310.09424/) (2023)
- [Adaptive-Control-Oriented Meta-Learning for Nonlinear Systems](https://mlsanity.com/paper/2103.04490/) (2021)
- [Deep networks for system identification: a Survey](https://mlsanity.com/paper/2301.12832/) (2023)
- [Continuous-time system identification with neural networks: Model structures and fitting criteria](https://mlsanity.com/paper/2006.02915/) (2020)
- [Nonlinear state-space identification using deep encoder networks](https://mlsanity.com/paper/2012.07697/) (2020)
- [dynoNet: a neural network architecture for learning dynamical systems](https://mlsanity.com/paper/2006.02250/) (2020)
- [Fast Online Adaptation in Robotics through Meta-Learning Embeddings of Simulated Priors](https://mlsanity.com/paper/2003.04663/) (2020)
- [Can Transformers Learn Optimal Filtering for Unknown Systems?](https://mlsanity.com/paper/2308.08536/) (2023)
- [From system models to class models: An in-context learning paradigm](https://mlsanity.com/paper/2308.13380/) (2023)
- [Meta-Learning of Neural State-Space Models Using Data From Similar Systems](https://mlsanity.com/paper/2211.07768/) (2022)
- [InCoRo: In-Context Learning for Robotics Control with Feedback Loops](https://mlsanity.com/paper/2402.05188/) (2024)
- [Sparse Bayesian Deep Learning for Dynamic System Identification](https://mlsanity.com/paper/2107.12910/) (2021)
- [Baseline Results for Selected Nonlinear System Identification Benchmarks](https://mlsanity.com/paper/2405.10779/) (2024)
- [On the adaptation of in-context learners for system identification](https://mlsanity.com/paper/2312.04083/) (2023)
- [Enhanced Transformer architecture for in-context learning of dynamical systems](https://mlsanity.com/paper/2410.03291/) (2024)
- [Meta-Learning for Physically-Constrained Neural System Identification](https://mlsanity.com/paper/2501.06167/) (2025)
- [Synthetic data generation for system identification: leveraging knowledge transfer from similar systems](https://mlsanity.com/paper/2403.05164/) (2024)
- [Distributionally robust minimization in meta-learning for system identification](https://mlsanity.com/paper/2506.18074/) (2025)
- [Successive convex optimization for transformer encoder model predictive control](https://mlsanity.com/paper/2605.14846/) (2026)

## Abstract (arXiv)

Traditional system identification typically involves two steps: initially, training data is processed to estimate a model of the system of interest; then, the model is used to predict outputs for new input queries. In contrast, recent in-context learning approaches learn a meta model that maps directly from training data and query inputs to outputs in a zero-shot fashion, bypassing explicit model estimation. This approach leverages a meta model trained offline, on large sets of synthetically generated data, to emulate the behavior of a system identification algorithm. In this work, we advance this meta-learning framework for system identification in three directions. First, we develop a unified probabilistic meta-identification architecture that combines a Transformer-based encoder-decoder with tailored mechanisms for handling long training sequences, enabling calibrated predictions over long horizons. Second, we show how a pre-trained meta model can act as an uncertainty-aware synthetic data generator for classical system identification, providing task-specific data that regularize parametric models when real data are scarce. Third, we systematically evaluate the framework on standard nonlinear system identification benchmarks and on a challenging out-of-distribution scenario, demonstrating competitive accuracy with state-of-the-art methods and strong generalization, including cases where input-only adaptation is sufficient to handle significant distribution shifts.