mlsanityData to 7 Oct 2026

Mathematical Formalism for Memory Compression in Selective State Space Models

arXiv 2410.03158 ↗First posted 4 Oct 2024cs.LG cs.AI cs.CCcited by 2

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

State space models (SSMs) have emerged as a powerful framework for modelling long-range dependencies in sequence data. Unlike traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs), SSMs offer a structured and stable approach to sequence modelling, leveraging principles from control theory and dynamical systems. However, a key challenge in sequence modelling is compressing long-term dependencies into a compact hidden state representation without losing critical information. In this paper, we develop a rigorous mathematical framework for understanding memory compression in selective state space models. We introduce a selective gating mechanism that dynamically filters and updates the hidden state based on input relevance, allowing for efficient memory compression. We formalize the trade-off between memory efficiency and information retention using information-theoretic tools, such as mutual information and rate-distortion theory. Our analysis provides theoretical bounds on the amount of information that can be compressed without sacrificing model performance. We also derive theorems that prove the stability and convergence of the hidden state in selective SSMs, ensuring reliable long-term memory retention. Computational complexity analysis reveals that selective SSMs offer significant improvements in memory efficiency and processing speed compared to traditional RNN-based models. Through empirical validation on sequence modelling tasks such as time-series forecasting and natural language processing, we demonstrate that selective SSMs achieve state-of-the-art performance while using less memory and computational resources.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 28 of its 50 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

28 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  2. 2024Theoretical Foundations of Deep Selective State-Space Models29 Feb
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  4. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  5. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 23

  1. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  2. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  3. 2024Motion Mamba: Efficient and Long Sequence Motion Generationcited by 111
  4. 2019Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inferencecited by 91
  5. 2021MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learningcited by 87
  6. 2020Learning to Continually Learncited by 82
  7. 2020Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compressioncited by 72
  8. 2020Bayesian Experimental Design for Implicit Models by Mutual Information Neural Estimationcited by 56
  9. 2021Entropy-based Logic Explanations of Neural Networkscited by 51
  10. 2021Towards a Unified Information-Theoretic Framework for Generalizationcited by 29
  11. 2021A comparison of LSTM and GRU networks for learning symbolic sequencescited by 24
  12. 2021Lipschitz Continuity Guided Knowledge Distillationcited by 19
11 more
  1. 2019Self-Attention Enhanced Selective Gate with Entity-Aware Embedding for Distantly Supervised Relation Extractioncited by 18
  2. 2021Neural Network Training Using ℓ1-Regularization and Bi-fidelity Datacited by 16
  3. 2019Gating Revisited: Deep Multi-layer RNNs That Can Be Trainedcited by 15
  4. 2024DyGMamba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Modelscited by 14
  5. 2021Long Short-term Memory RNNcited by 10
  6. 2019Mutual Information Maximization in Graph Neural Networkscited by 9
  7. 2022Neural Estimation of the Rate-Distortion Function With Applications to Operational Source Codingcited by 9
  8. 2020Precision Gating: Improving Neural Network Efficiency with Dynamic Dual-Precision Activationscited by 5
  9. 2023Towards Efficient In-memory Computing Hardware for Quantized Neural Networks: State-of-the-art, Open Challenges and Perspectivescited by 5
  10. 2020Demystifying Deep Learning in Predictive Spatio-Temporal Analytics: An Information-Theoretic Frameworkcited by 2
  11. 2022Rethinking Fano's Inequality in Ensemble Learningcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  2. 2025MambaX: Image Super-Resolution with State Predictive Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Curious Case of In-Training Compression of State Space Models
  2. 2025Rethinking Selectivity in State Space Models: A Minimal Predictive Sufficiency Approach
  3. 2026On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
  4. 2025SeRpEnt: Selective Resampling for Expressive State Space Models
  5. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory
  6. 2025Characterizing Mamba's Selective Memory using Auto-Encoders
  7. 2026A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Models
  8. 2026Sessa: Selective State Space Attention