# Spectral State Space Models

arXiv: https://arxiv.org/abs/2312.06837 · HTML: https://mlsanity.com/paper/2312.06837/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Hungry Hungry Hippos: Towards Language Modeling with State Space Models](https://mlsanity.com/paper/2212.14052/) (2022)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [It's Raw! Audio Generation with State-Space Models](https://mlsanity.com/paper/2202.09729/) (2022)
- [Learning Without Mixing: Towards A Sharp Analysis of Linear System Identification](https://mlsanity.com/paper/1802.08334/) (2018)
- [What Makes Convolutional Models Great on Long Sequence Modeling?](https://mlsanity.com/paper/2210.09298/) (2022)
- [Simple Hardware-Efficient Long Convolutions for Sequence Modeling](https://mlsanity.com/paper/2302.06646/) (2023)
- [Effectively Modeling Time Series with Simple Discrete State Spaces](https://mlsanity.com/paper/2303.09489/) (2023)
- [Learning Linear Dynamical Systems via Spectral Filtering](https://mlsanity.com/paper/1711.00946/) (2017)
- [Spectral Filtering for General Linear Dynamical Systems](https://mlsanity.com/paper/1802.03981/) (2018)
- [Introduction to Online Control](https://mlsanity.com/paper/2211.09619/) (2022)
- [Sequence Modeling with Multiresolution Convolutional Memory](https://mlsanity.com/paper/2305.01638/) (2023)
- [No-Regret Prediction in Marginally Stable Systems](https://mlsanity.com/paper/2002.02064/) (2020)

## Built on by (most cited first, 21 of 21)

- [ZigMa: A DiT-style Zigzag Mamba Diffusion Model](https://mlsanity.com/paper/2403.13802/) (2024)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [ThunderKittens: Simple, Fast, and Adorable AI Kernels](https://mlsanity.com/paper/2410.20399/) (2024)
- [Tuning Frequency Bias of State Space Models](https://mlsanity.com/paper/2410.02035/) (2024)
- [HOPE for a Robust Parameterization of Long-memory State Space Models](https://mlsanity.com/paper/2405.13975/) (2024)
- [Sequences of Logits Reveal the Low Rank Structure of Language Models](https://mlsanity.com/paper/2510.24966/) (2025)
- [Block-Biased Mamba for Long-Range Sequence Processing](https://mlsanity.com/paper/2505.09022/) (2025)
- [Flash STU: Fast Spectral Transform Units](https://mlsanity.com/paper/2409.10489/) (2024)
- [FutureFill: Fast Generation from Convolutional Sequence Models](https://mlsanity.com/paper/2410.03766/) (2024)
- [Universal Sequence Preconditioning](https://mlsanity.com/paper/2502.06545/) (2025)
- [Universal Learning of Nonlinear Dynamics](https://mlsanity.com/paper/2508.11990/) (2025)
- [Provable Length Generalization in Sequence Prediction via Spectral Filtering](https://mlsanity.com/paper/2411.01035/) (2024)
- [A Deep State Space Model for Rainfall-Runoff Simulations](https://mlsanity.com/paper/2501.14980/) (2025)
- [MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling](https://mlsanity.com/paper/2512.23824/) (2025)
- [Elastic Spectral State Space Models for Train-Once Budgeted Inference](https://mlsanity.com/paper/2601.22488/) (2026)
- [LISM: Long-range Integrative State space Models via Input-Latent State Interactions](https://mlsanity.com/paper/2509.04226/) (2025)
- [Aligning Inductive Bias for Data-Efficient Generalization in State Space Models](https://mlsanity.com/paper/2509.20789/) (2025)
- [PDE-SSM: A Spectral State Space Approach to Spatial Mixing in Diffusion Transformers](https://mlsanity.com/paper/2603.13663/) (2026)
- [A Spectral Filtering Approach to Regret Analysis of Distributed Online Control for Linear Dynamical Systems](https://mlsanity.com/paper/2608.02375/) (2026)
- [Elastic Selective Spectral Hybrids for Train-Once, Export-Many Budgeted Inference](https://mlsanity.com/paper/2609.32486/) (2026)
- [MS-GLA: Multi-Scale Gated Linear Attention for Addressing Representational Bottlenecks via Multi-Temporal Resolution](https://mlsanity.com/paper/2609.35664/) (2026)

## Abstract (arXiv)

This paper studies sequence modeling for prediction tasks with long range dependencies. We propose a new formulation for state space models (SSMs) based on learning linear dynamical systems with the spectral filtering algorithm (Hazan et al. (2017)). This gives rise to a novel sequence prediction architecture we call a spectral state space model. Spectral state space models have two primary advantages. First, they have provable robustness properties as their performance depends on neither the spectrum of the underlying dynamics nor the dimensionality of the problem. Second, these models are constructed with fixed convolutional filters that do not require learning while still outperforming SSMs in both theory and practice. The resulting models are evaluated on synthetic dynamical systems and long-range prediction tasks of various modalities. These evaluations support the theoretical benefits of spectral filtering for tasks requiring very long range memory.