mlsanityData to 7 Oct 2026

Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers

arXiv 2110.13985 ↗First posted 26 Oct 2021cs.LG cs.AIcited by 607

Not on the map. Why?

Cited by 305 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recurrent neural networks (RNNs), temporal convolutions, and neural differential equations (NDEs) are popular families of deep learning models for time-series data, each with unique strengths and tradeoffs in modeling power and computational efficiency. We introduce a simple sequence model inspired by control systems that generalizes these approaches while addressing their shortcomings. The Linear State-Space Layer (LSSL) maps a sequence u mapsto y by simply simulating a linear continuous-time state-space representation dotx = Ax + Bu, y = Cx + Du. Theoretically, we show that LSSL models are closely related to the three aforementioned families of models and inherit their strengths. For example, they generalize convolutions to continuous-time, explain common RNN heuristics, and share features of NDEs such as time-scale adaptation. We then incorporate and generalize recent theory on continuous-time memorization to introduce a trainable subset of structured matrices A that endow LSSLs with long-range memory. Empirically, stacking LSSL layers into a simple deep neural network obtains state-of-the-art results across time series benchmarks for long dependencies in sequential image classification, real-world healthcare regression tasks, and speech. On a difficult speech classification task with length-16000 sequences, LSSL outperforms prior approaches by 24 accuracy points, and even outperforms baselines that use hand-crafted features on 100x shorter sequences.

Key facts. Cited by 607 papers in the map (data through 7 Oct 2026). Builds on 34 of its 65 references that are in the map. First posted on arXiv 26 Oct 2021.

Builds on

34 of 65 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  6. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  7. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  8. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  9. 2016Recurrent Neural Networks for Multivariate Time Series with Missing Valuescited by 548
  10. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  11. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  12. 2019GRU-ODE-Bayes: Continuous modeling of sporadically-observed time seriescited by 230
22 more
  1. 2020Understanding the Difficulty of Training Transformerscited by 210
  2. 2020Liquid Time-constant Networkscited by 160
  3. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 142
  4. 2020Neural Rough Differential Equations for Long Time Seriescited by 140
  5. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  6. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  7. 2020Lipschitz Recurrent Neural Networkscited by 88
  8. 2016Quasi-Recurrent Neural Networkscited by 87
  9. 2017Dilated Recurrent Neural Networkscited by 83
  10. 2020Learning Long-Term Dependencies in Irregularly-Sampled Time Seriescited by 82
  11. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 81
  12. 2018The Temple University Hospital Seizure Detection Corpuscited by 80
  13. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  14. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 60
  15. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  16. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  17. 2018Can recurrent neural networks warp time?cited by 53
  18. 2018Trellis Networks for Sequence Modelingcited by 50
  19. 2018The PyTorch-Kaldi Speech Recognition Toolkitcited by 38
  20. 2021Parallelizing Legendre Memory Unit Trainingcited by 22
  21. 2019Gated recurrent units viewed through the lens of continuous time dynamical systemscited by 18
  22. 2019Recurrent Neural Networks in the Eye of Differential Equationscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 21; 2023: 41; 2024: 281; 2025: 183; 2026: 81.

607 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  3. 2024VMamba: Visual State Space Modelcited by 955
  4. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  5. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  6. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  7. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  8. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  9. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  10. 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 385
  11. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  12. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 349
38 more
  1. 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 286
  2. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
  3. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  4. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  5. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  6. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 191
  7. 2024The Illusion of State in State-Space Modelscited by 181
  8. 2024Swin-UMamba: Mamba-based UNet with ImageNet-based pretrainingcited by 179
  9. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  10. 2022FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecastingcited by 147
  11. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mambacited by 146
  12. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 142
  13. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  14. 2022Liquid Structural State-Space Modelscited by 134
  15. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  16. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  17. 2024MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detectioncited by 113
  18. 2024Motion Mamba: Efficient and Long Sequence Motion Generationcited by 111
  19. 2024Is Mamba Effective for Time Series Forecasting?cited by 110
  20. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  21. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  22. 2022Long Movie Clip Classification with State-Space Video Modelscited by 106
  23. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  24. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  25. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  26. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  27. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  28. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 97
  29. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  30. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  31. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  32. 2024VmambaIR: Visual State Space Model for Image Restorationcited by 90
  33. 2024The Hidden Attention of Mamba Modelscited by 88
  34. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 86
  35. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85
  36. 2024Graph Mamba: Towards Learning on Graphs with State Space Modelscited by 84
  37. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  38. 2024FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mambacited by 81

50 most cited of 607 All 607 (JSON)