mlsanityData to 7 Oct 2026

HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context

arXiv 2407.09375 ↗First posted 12 Jul 2024cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

This work explores the in-context learning capabilities of State Space Models (SSMs) and presents, to the best of our knowledge, the first theoretical explanation of a possible underlying mechanism. We introduce a novel weight construction for SSMs, enabling them to predict the next state of any dynamical system after observing previous states without parameter fine-tuning. This is accomplished by extending the HiPPO framework to demonstrate that continuous SSMs can approximate the derivative of any input signal. Specifically, we find an explicit weight construction for continuous SSMs and provide an asymptotic error bound on the derivative approximation. The discretization of this continuous SSM subsequently yields a discrete SSM that predicts the next state. Finally, we demonstrate the effectiveness of our parameterization empirically. This work should be an initial step toward understanding how sequence models based on SSMs learn in context.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 15 of its 20 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

15 of 20 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  4. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  5. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  6. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  7. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  8. 2023Larger language models do in-context learning differentlycited by 386
  9. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  10. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  11. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  12. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 168
3 more
  1. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 121
  2. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  3. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025From Markov to Laplace: How Mamba In-Context Learns Markov Chainscited by 9
  2. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  3. 2025How Can Mamba Learn In Context with Outliers and Generalize Provably?cited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025UnHiPPO: Uncertainty-aware Initialization for State Space Models
  2. 2024Deep Learning-based Approaches for State Space Models: A Selective Review
  3. 2022Hidden Parameter Recurrent State Space Models For Changing Dynamics Scenarios
  4. 2025The Curious Case of In-Training Compression of State Space Models
  5. 2024Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models
  6. 2026MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models
  7. 2025Rethinking Selectivity in State Space Models: A Minimal Predictive Sufficiency Approach
  8. 2025Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models