mlsanityData to 7 Oct 2026

Liquid Structural State-Space Models

arXiv 2209.12951 ↗First posted 26 Sep 2022cs.LG cs.AI cs.CLcited by 134

Not on the map. Why?

Cited by 31 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A proper parametrization of state transition matrices of linear state-space models (SSMs) followed by standard nonlinearities enables them to efficiently learn representations from sequential data, establishing the state-of-the-art on a large series of long-range sequence modeling benchmarks. In this paper, we show that we can improve further when the structural SSM such as S4 is given by a linear liquid time-constant (LTC) state-space model. LTC neural networks are causal continuous-time neural networks with an input-dependent state transition module, which makes them learn to adapt to incoming inputs at inference. We show that by using a diagonal plus low-rank decomposition of the state transition matrix introduced in S4, and a few simplifications, the LTC-based structural state-space model, dubbed Liquid-S4, achieves the new state-of-the-art generalization across sequence modeling tasks with long-term dependencies such as image, text, audio, and medical time-series, with an average performance of 87.32% on the Long-Range Arena benchmark. On the full raw Speech Command recognition, dataset Liquid-S4 achieves 96.78% accuracy with a 30% reduction in parameter counts compared to S4. The additional gain in performance is the direct result of the Liquid-S4's kernel structure that takes into account the similarities of the input sequence samples during training and inference.

Key facts. Cited by 134 papers in the map (data through 7 Oct 2026). Builds on 66 of its 102 references that are in the map. First posted on arXiv 26 Sep 2022.

Builds on

66 of 102 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Neural Ordinary Differential Equationscited by 3,733
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2020Rethinking Attention with Performerscited by 1,584
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2022Simplified State Space Layers for Sequence Modelingcited by 776
38 more
  1. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  2. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  3. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  4. 2019Neural Spline Flowscited by 590
  5. 2016Recurrent Neural Networks for Multivariate Time Series with Missing Valuescited by 548
  6. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  7. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  8. 2020Hopfield Networks is All You Needcited by 461
  9. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  10. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 415
  11. 2019Augmented Neural ODEscited by 396
  12. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  13. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  14. 2015LSTM: A Search Space Odysseycited by 315
  15. 2020Sparse Sinkhorn Attentioncited by 250
  16. 2019GRU-ODE-Bayes: Continuous modeling of sporadically-observed time seriescited by 230
  17. 2016Temporal Convolutional Networks: A Unified Approach to Action Segmentationcited by 199
  18. 2020Liquid Time-constant Networkscited by 160
  19. 2020Dissecting Neural ODEscited by 150
  20. 2019Neural Jump Stochastic Differential Equationscited by 147
  21. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 142
  22. 2020Neural Rough Differential Equations for Long Time Seriescited by 140
  23. 2020Learning to Control PDEs with Differentiable Physicscited by 131
  24. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 121
  25. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  26. 2020Kernel Operations on the GPU, with Autodiff, without Memory Overflowscited by 101
  27. 2016Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequencescited by 98
  28. 2021Luna: Linear Unified Nested Attentioncited by 96
  29. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 90
  30. 2020Lipschitz Recurrent Neural Networkscited by 88
  31. 2017Dilated Recurrent Neural Networkscited by 83
  32. 2019On Robustness of Neural Ordinary Differential Equationscited by 83
  33. 2020Learning Long-Term Dependencies in Irregularly-Sampled Time Seriescited by 82
  34. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 81
  35. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  36. 2021Closed-form Continuous-time Neural Modelscited by 65
  37. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  38. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 18; 2024: 47; 2025: 43; 2026: 22.

134 papers.

Sort:

By area:

  1. 2024VMamba: Visual State Space Modelcited by 955
  2. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  3. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  4. 2024Titans: Learning to Memorize at Test Timecited by 303
  5. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  6. 2024The Illusion of State in State-Space Modelscited by 181
  7. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  8. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  9. 2024RS-Mamba for Large Remote Sensing Image Dense Predictioncited by 80
  10. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 71
  11. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  12. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
38 more
  1. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  2. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  3. 2024Retinexmamba: Retinex-based Mamba for Low-light Image Enhancementcited by 59
  4. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  5. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  6. 2023Dataset Distillation with Convexified Implicit Gradientscited by 46
  7. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  8. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  9. 2024Scalable Diffusion Models with State Space Backbonecited by 45
  10. 2024Were RNNs All We Needed?cited by 36
  11. 2024VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecastingcited by 35
  12. 2024Oscillatory State-Space Modelscited by 35
  13. 2024ThunderKittens: Simple, Fast, and Adorable AI Kernelscited by 34
  14. 2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32
  15. 2024B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memorycited by 26
  16. 2024Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERTcited by 21
  17. 2023Sparse Modular Activation for Efficient Sequence Modelingcited by 20
  18. 2024MetaLA: Unified Optimal Linear Approximation to Softmax Attention Mapcited by 20
  19. 2023Sequence Modeling with Multiresolution Convolutional Memorycited by 19
  20. 2023Learning Long Sequences in Spiking Neural Networkscited by 19
  21. 2023Robustifying State-space Models for Long Sequences via Approximate Diagonalizationcited by 18
  22. 2024State-Free Inference of State-Space Models: The Transfer Function Approachcited by 18
  23. 2025Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generationcited by 17
  24. 2024Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motioncited by 16
  25. 2024Tuning Frequency Bias of State Space Modelscited by 16
  26. 2024MMR-Mamba: Multi-Modal MRI Reconstruction with Mamba and Spatial-Frequency Information Fusioncited by 15
  27. 2024GroupMamba: Efficient Group-Based Visual State Space Modelcited by 15
  28. 2024TinyViM: Frequency Decoupling for Tiny Hybrid Vision Mambacited by 15
  29. 2025Structured Sparse Transition Matrices to Enable State Tracking in State-Space Modelscited by 15
  30. 2023Advancing Regular Language Reasoning in Linear Recurrent Neural Networkscited by 13
  31. 2024GrootVL: Tree Topology is All You Need in State Space Modelcited by 13
  32. 2024What Can We Learn from State Space Models for Machine Learning on Graphs?cited by 13
  33. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  34. 2023Revisiting Structured Variational Autoencoderscited by 11
  35. 2023Recurrent Distance Filtering for Graph Representation Learningcited by 11
  36. 2024HOPE for a Robust Parameterization of Long-memory State Space Modelscited by 11
  37. 2024MaIL: Improving Imitation Learning with Mambacited by 11
  38. 2025Prototype-Driven Structure Synergy Network for Remote Sensing Images Segmentationcited by 11

50 most cited of 134 All 134 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026A Simple State Space Model Excels at Multivariate Time Series Classification
  2. 2023A Neural State-Space Model Approach to Efficient Speech Separation
  3. 2022Simplifying and Understanding State Space Models with Diagonal Linear RNNs
  4. 2026An explicit operator explains end-to-end computation in the modern neural networks used for sequence and language modeling
  5. 2024State Space Models as Foundation Models: A Control Theoretic Overview
  6. 2026On the Relation of State Space Models and Hidden Markov Models
  7. 2024Deep Learning-based Approaches for State Space Models: A Selective Review
  8. 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms