mlsanityData to 7 Oct 2026

Efficiently Modeling Long Sequences with Structured State Spaces

arXiv 2111.00396 ↗First posted 31 Oct 2021cs.LGcited by 2,424

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

A central goal of sequence modeling is designing a single principled model that can address sequence data across a range of modalities and tasks, particularly on long-range dependencies. Although conventional models including RNNs, CNNs, and Transformers have specialized variants for capturing long dependencies, they still struggle to scale to very long sequences of 10000 or more steps. A promising recent approach proposed modeling sequences by simulating the fundamental state space model (SSM) ( x'(t) = Ax(t) + Bu(t), y(t) = Cx(t) + Du(t) ), and showed that for appropriate choices of the state matrix ( A ), this system could handle long-range dependencies mathematically and empirically. However, this method has prohibitive computation and memory requirements, rendering it infeasible as a general sequence modeling solution. We propose the Structured State Space sequence model (S4) based on a new parameterization for the SSM, and show that it can be computed much more efficiently than prior approaches while preserving their theoretical strengths. Our technique involves conditioning ( A ) with a low-rank correction, allowing it to be diagonalized stably and reducing the SSM to the well-studied computation of a Cauchy kernel. S4 achieves strong empirical results across a diverse range of established benchmarks, including (i) 91% accuracy on sequential CIFAR-10 with no data augmentation or auxiliary losses, on par with a larger 2-D ResNet, (ii) substantially closing the gap to Transformers on image and language modeling tasks, while performing generation 60× faster (iii) SoTA on every task from the Long Range Arena benchmark, including solving the challenging Path-X task of length 16k that all prior work fails on, while being as efficient as all competitors.

Key facts. Cited by 2,424 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 33 of its 61 references that are in the map. First posted on arXiv 31 Oct 2021.

Builds on

33 of 61 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  10. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  11. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  12. 2016Language Modeling with Gated Convolutional Networkscited by 688
21 more
  1. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  2. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  3. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  4. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  5. 2019GRU-ODE-Bayes: Continuous modeling of sporadically-observed time seriescited by 230
  6. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  7. 2018Adversarial Audio Synthesiscited by 222
  8. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
  9. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 142
  10. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  11. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  12. 2020Lipschitz Recurrent Neural Networkscited by 88
  13. 2017Dilated Recurrent Neural Networkscited by 83
  14. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  15. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  16. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  17. 2018Trellis Networks for Sequence Modelingcited by 50
  18. 2021Parallelizing Legendre Memory Unit Trainingcited by 22
  19. 2017Fast Generation for Convolutional Autoregressive Modelscited by 14
  20. 2018Fast Parametric Learning with Activation Memorizationcited by 9
  21. 2020Time-aware Large Kernel Convolutionscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 63; 2023: 171; 2024: 805; 2025: 750; 2026: 634.

2,424 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  6. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  7. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,030
  8. 2024VMamba: Visual State Space Modelcited by 955
  9. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  10. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  11. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  12. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
38 more
  1. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  2. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  3. 2024Large Language Models: A Surveycited by 506
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  5. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  6. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  7. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  8. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  9. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  10. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  11. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  13. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  14. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  15. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 349
  16. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  17. 2023Long-term Forecasting with TiDE: Time-series Dense Encodercited by 327
  18. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  19. 2024Titans: Learning to Memorize at Test Timecited by 303
  20. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  21. 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 286
  22. 2022Long Range Language Modeling via Gated State Spacescited by 265
  23. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  24. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
  25. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  26. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 232
  27. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  28. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  29. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  30. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  31. 2023PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solverscited by 205
  32. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  33. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  34. 2022Diffusion-based Time Series Imputation and Forecasting with Structured State Space Modelscited by 195
  35. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  36. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  37. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 191
  38. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 187

50 most cited of 2,424 All 2,424 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces
  2. 2023Convolutional State Space Models for Long-Range Spatiotemporal Modeling
  3. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory
  4. 2024State-Space Modeling in Long Sequence Processing: A Survey on Recurrence in the Transformer Era
  5. 2025SequenceLayers: Sequence Processing and Streaming Neural Networks Made Easy
  6. 2025SeRpEnt: Selective Resampling for Expressive State Space Models
  7. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues
  8. 2026Benchmarking the Computational and Representational Efficiency of State Space Models against Transformers on Long-Context Dyadic Sessions