# State Space Models as Foundation Models: A Control Theoretic Overview

arXiv: https://arxiv.org/abs/2403.16899 · HTML: https://mlsanity.com/paper/2403.16899/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Hungry Hungry Hippos: Towards Language Modeling with State Space Models](https://mlsanity.com/paper/2212.14052/) (2022)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models](https://mlsanity.com/paper/2402.19427/) (2024)
- [It's Raw! Audio Generation with State-Space Models](https://mlsanity.com/paper/2202.09729/) (2022)
- [The Hidden Attention of Mamba Models](https://mlsanity.com/paper/2403.01590/) (2024)

## Built on by (most cited first, 19 of 19)

- [HGRN2: Gated Linear RNNs with State Expansion](https://mlsanity.com/paper/2404.07904/) (2024)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networks](https://mlsanity.com/paper/2405.15731/) (2024)
- [Latent Mamba Operator for Partial Differential Equations](https://mlsanity.com/paper/2505.19105/) (2025)
- [Block-Biased Mamba for Long-Range Sequence Processing](https://mlsanity.com/paper/2505.09022/) (2025)
- [Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Tracking](https://mlsanity.com/paper/2602.10743/) (2026)
- [State Space Models are Provably Comparable to Transformers in Dynamic Token Selection](https://mlsanity.com/paper/2405.19036/) (2024)
- [Toward Physics-guided Time Series Embedding](https://mlsanity.com/paper/2410.06651/) (2024)
- [The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labels](https://mlsanity.com/paper/2410.10473/) (2024)
- [Learning in the Recurrent State: Gradient Descent with Linear Recurrent Networks](https://mlsanity.com/paper/2410.11687/) (2024)
- [State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?](https://mlsanity.com/paper/2409.13421/) (2024)
- [Context-Selective State Space Models: Feedback is All You Need](https://mlsanity.com/paper/2510.14027/) (2025)
- [A Unified Framework for Neural Computation and Learning Over Time](https://mlsanity.com/paper/2409.12038/) (2024)
- [L2RU: a Structured State Space Model with prescribed L2-bound](https://mlsanity.com/paper/2503.23818/) (2025)
- [Controller Design for Structured State-space Models via Contraction Theory](https://mlsanity.com/paper/2604.07069/) (2026)
- [Towards Understanding Self-Pretraining for Sequence Classification](https://mlsanity.com/paper/2605.21070/) (2026)
- [Observability conditions for neural state-space models with eigenvalues and their roots of unity](https://mlsanity.com/paper/2504.15758/) (2025)
- [Continuity Laws for Sequential Models](https://mlsanity.com/paper/2605.08539/) (2026)
- [Deep learning from a control vantage point](https://mlsanity.com/paper/2610.14385/) (2026)

## Abstract (arXiv)

In recent years, there has been a growing interest in integrating linear state-space models (SSM) in deep neural network architectures of foundation models. This is exemplified by the recent success of Mamba, showing better performance than the state-of-the-art Transformer architectures in language tasks. Foundation models, like e.g. GPT-4, aim to encode sequential data into a latent space in order to learn a compressed representation of the data. The same goal has been pursued by control theorists using SSMs to efficiently model dynamical systems. Therefore, SSMs can be naturally connected to deep sequence modeling, offering the opportunity to create synergies between the corresponding research areas. This paper is intended as a gentle introduction to SSM-based architectures for control theorists and summarizes the latest research developments. It provides a systematic review of the most successful SSM proposals and highlights their main features from a control theoretic perspective. Additionally, we present a comparative analysis of these models, evaluating their performance on a standardized benchmark designed for assessing a model's efficiency at learning long sequences.