mlsanityData to 14 Oct 2026

Simplified State Space Layers for Sequence Modeling

arXiv 2208.04933 ↗First posted 9 Aug 2022cs.LGcited by 788

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Models using structured state space sequence (S4) layers have achieved state-of-the-art performance on long-range sequence modeling tasks. An S4 layer combines linear state space models (SSMs), the HiPPO framework, and deep learning to achieve high performance. We build on the design of the S4 layer and introduce a new state space layer, the S5 layer. Whereas an S4 layer uses many independent single-input, single-output SSMs, the S5 layer uses one multi-input, multi-output SSM. We establish a connection between S5 and S4, and use this to develop the initialization and parameterization used by the S5 model. The result is a state space layer that can leverage efficient and widely implemented parallel scans, allowing S5 to match the computational efficiency of S4, while also achieving state-of-the-art performance on several long-range sequence modeling tasks. S5 averages 87.4% on the long range arena benchmark, and 98.5% on the most difficult Path-X task.

Key facts. Cited by 788 papers in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 55 of its 83 references that are in the map. First posted on arXiv 9 Aug 2022.

Builds on

55 of 83 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections24 Jun
  2. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  3. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2018Neural Ordinary Differential Equationscited by 3,779
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  9. 2020Reformer: The Efficient Transformercited by 1,668
  10. 2020Rethinking Attention with Performerscited by 1,601
  11. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
38 more
  1. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  2. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  3. 2016Language Modeling with Gated Convolutional Networkscited by 702
  4. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  5. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  6. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  7. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  8. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  9. 2021FNet: Mixing Tokens with Fourier Transformscited by 347
  10. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  11. 2021Multi-Time Attention Networks for Irregularly Sampled Time Seriescited by 238
  12. 2019GRU-ODE-Bayes: Continuous modeling of sporadically-observed time seriescited by 231
  13. 2018Adversarial Audio Synthesiscited by 225
  14. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  15. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 143
  16. 2020Neural Rough Differential Equations for Long Time Seriescited by 143
  17. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
  18. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 129
  19. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 127
  20. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 120
  21. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  22. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  23. 2021Modeling Irregular Time Series with Continuous Recurrent Unitscited by 103
  24. 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98
  25. 2021Luna: Linear Unified Nested Attentioncited by 96
  26. 2020Lipschitz Recurrent Neural Networkscited by 89
  27. 2016Quasi-Recurrent Neural Networkscited by 87
  28. 2017Dilated Recurrent Neural Networkscited by 83
  29. 2019Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spacescited by 67
  30. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  31. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 61
  32. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  33. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  34. 2018Trellis Networks for Sequence Modelingcited by 52
  35. 2021H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequencescited by 47
  36. 2020GMAT: Global Memory Augmentation for Transformerscited by 36
  37. 2019Temporal Parallelization of Bayesian Smootherscited by 33
  38. 2021Parallelizing Legendre Memory Unit Trainingcited by 23

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 62; 2024: 283; 2025: 245; 2026: 194.

788 papers.

Sort:

By area:

  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  2. 2024VMamba: Visual State Space Modelcited by 969
  3. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  4. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  5. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  6. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  7. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 352
  8. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  9. 2024Titans: Learning to Memorize at Test Timecited by 328
  10. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  11. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 236
  12. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
38 more
  1. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  2. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 194
  3. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
  4. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 170
  5. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 167
  6. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mambacited by 146
  7. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  8. 2022Liquid Structural State-Space Modelscited by 136
  9. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  10. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  11. 2024ChangeMamba: Remote Sensing Change Detection With Spatiotemporal State Space Modelcited by 120
  12. 2024Is Mamba Effective for Time Series Forecasting?cited by 114
  13. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 113
  14. 2024MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detectioncited by 113
  15. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  16. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 108
  17. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 103
  18. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  19. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 98
  20. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  21. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  22. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 92
  23. 2024The Hidden Attention of Mamba Modelscited by 91
  24. 2024VmambaIR: Visual State Space Model for Image Restorationcited by 90
  25. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 89
  26. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  27. 2024RS-Mamba for Large Remote Sensing Image Dense Predictioncited by 81
  28. 2024State Space Models for Event Camerascited by 75
  29. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 75
  30. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  31. 2024Pan-Mamba: Effective pan-sharpening with State Space Modelcited by 73
  32. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 73
  33. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesiscited by 72
  34. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 72
  35. 2024Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentationcited by 71
  36. 2024Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detectioncited by 66
  37. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 66
  38. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 64

50 most cited of 788 All 788 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling
  2. 2022What Makes Convolutional Models Great on Long Sequence Modeling?
  3. 2026A Simple State Space Model Excels at Multivariate Time Series Classification
  4. 2025SequenceLayers: Sequence Processing and Streaming Neural Networks Made Easy
  5. 2026S4oP: Operator-level Pruning of Structured State Space Models for Resource-Constrained Devices
  6. 2024State Space Models are Provably Comparable to Transformers in Dynamic Token Selection
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces
  8. 2025From Layers to States: A State Space Model Perspective to Deep Neural Network Layer Dynamics