mlsanityData to 7 Oct 2026

Enhanced Transformer architecture for in-context learning of dynamical systems

arXiv 2410.03291 ↗First posted 4 Oct 2024cs.LG cs.AI cs.SYcited by 4

Not on the map. Why?

Abstract (arXiv)

Recently introduced by some of the authors, the in-context identification paradigm aims at estimating, offline and based on synthetic data, a meta-model that describes the behavior of a whole class of systems. Once trained, this meta-model is fed with an observed input/output sequence (context) generated by a real system to predict its behavior in a zero-shot learning fashion. In this paper, we enhance the original meta-modeling framework through three key innovations: by formulating the learning task within a probabilistic framework; by managing non-contiguous context and query windows; and by adopting recurrent patching to effectively handle long context sequences. The efficacy of these modifications is demonstrated through a numerical example focusing on the Wiener-Hammerstein system class, highlighting the model's enhanced performance and scalability.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

18 of 29 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  8. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  9. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  10. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  11. 2020dynoNet: a neural network architecture for learning dynamical systemscited by 28
  12. 2023Can Transformers Learn Optimal Filtering for Unknown Systems?cited by 15
6 more
  1. 2023From system models to class models: An in-context learning paradigmcited by 13
  2. 2022Deep Subspace Encoders for Nonlinear System Identificationcited by 12
  3. 2022Meta-Learning of Neural State-Space Models Using Data From Similar Systemscited by 11
  4. 2021Training Recurrent Neural Networks by Sequential Least Squares and the Alternating Direction Method of Multiplierscited by 8
  5. 2023On the adaptation of in-context learners for system identificationcited by 5
  6. 2024Synthetic data generation for system identification: leveraging knowledge transfer from similar systemscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Distributionally robust minimization in meta-learning for system identificationcited by 2
  2. 2026Diffusion Sequence Models for Generative In-Context Meta-Learning of Robot Dynamicscited by 0
  3. 2026Target Parameterization in Diffusion Models for Nonlinear Spatiotemporal System Identificationcited by 0
  4. 2026Variational meta-learning inference for low dimensional neural system identificationcited by 0