# Mathematical Formalism for Memory Compression in Selective State Space Models arXiv: https://arxiv.org/abs/2410.03158 · HTML: https://mlsanity.com/paper/2410.03158/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). - Builds on 28 of its 50 references that are in the map. - First posted on arXiv 4 Oct 2024. ## Placed in - State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/) ## Builds on (most cited first, 28 of 28) - [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023) - [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021) - [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024) - [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022) - [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018) - [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021) - [Motion Mamba: Efficient and Long Sequence Motion Generation](https://mlsanity.com/paper/2403.07487/) (2024) - [Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference](https://mlsanity.com/paper/1912.03203/) (2019) - [MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning](https://mlsanity.com/paper/2110.15352/) (2021) - [Learning to Continually Learn](https://mlsanity.com/paper/2002.09571/) (2020) - [Theoretical Foundations of Deep Selective State-Space Models](https://mlsanity.com/paper/2402.19047/) (2024) - [Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression](https://mlsanity.com/paper/2003.08935/) (2020) - [Bayesian Experimental Design for Implicit Models by Mutual Information Neural Estimation](https://mlsanity.com/paper/2002.08129/) (2020) - [Entropy-based Logic Explanations of Neural Networks](https://mlsanity.com/paper/2106.06804/) (2021) - [Towards a Unified Information-Theoretic Framework for Generalization](https://mlsanity.com/paper/2111.05275/) (2021) - [A comparison of LSTM and GRU networks for learning symbolic sequences](https://mlsanity.com/paper/2107.02248/) (2021) - [Lipschitz Continuity Guided Knowledge Distillation](https://mlsanity.com/paper/2108.12905/) (2021) - [Self-Attention Enhanced Selective Gate with Entity-Aware Embedding for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/1911.11899/) (2019) - [Neural Network Training Using ℓ1-Regularization and Bi-fidelity Data](https://mlsanity.com/paper/2105.13011/) (2021) - [Gating Revisited: Deep Multi-layer RNNs That Can Be Trained](https://mlsanity.com/paper/1911.11033/) (2019) - [DyGMamba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Models](https://mlsanity.com/paper/2408.04713/) (2024) - [Long Short-term Memory RNN](https://mlsanity.com/paper/2105.06756/) (2021) - [Mutual Information Maximization in Graph Neural Networks](https://mlsanity.com/paper/1905.08509/) (2019) - [Neural Estimation of the Rate-Distortion Function With Applications to Operational Source Coding](https://mlsanity.com/paper/2204.01612/) (2022) - [Precision Gating: Improving Neural Network Efficiency with Dynamic Dual-Precision Activations](https://mlsanity.com/paper/2002.07136/) (2020) - [Towards Efficient In-memory Computing Hardware for Quantized Neural Networks: State-of-the-art, Open Challenges and Perspectives](https://mlsanity.com/paper/2307.03936/) (2023) - [Demystifying Deep Learning in Predictive Spatio-Temporal Analytics: An Information-Theoretic Framework](https://mlsanity.com/paper/2009.06304/) (2020) - [Rethinking Fano's Inequality in Ensemble Learning](https://mlsanity.com/paper/2205.12683/) (2022) ## Built on by (most cited first, 2 of 2) - [Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba](https://mlsanity.com/paper/2503.18970/) (2025) - [MambaX: Image Super-Resolution with State Predictive Control](https://mlsanity.com/paper/2511.18028/) (2025) ## Abstract (arXiv) State space models (SSMs) have emerged as a powerful framework for modelling long-range dependencies in sequence data. Unlike traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs), SSMs offer a structured and stable approach to sequence modelling, leveraging principles from control theory and dynamical systems. However, a key challenge in sequence modelling is compressing long-term dependencies into a compact hidden state representation without losing critical information. In this paper, we develop a rigorous mathematical framework for understanding memory compression in selective state space models. We introduce a selective gating mechanism that dynamically filters and updates the hidden state based on input relevance, allowing for efficient memory compression. We formalize the trade-off between memory efficiency and information retention using information-theoretic tools, such as mutual information and rate-distortion theory. Our analysis provides theoretical bounds on the amount of information that can be compressed without sacrificing model performance. We also derive theorems that prove the stability and convergence of the hidden state in selective SSMs, ensuring reliable long-term memory retention. Computational complexity analysis reveals that selective SSMs offer significant improvements in memory efficiency and processing speed compared to traditional RNN-based models. Through empirical validation on sequence modelling tasks such as time-series forecasting and natural language processing, we demonstrate that selective SSMs achieve state-of-the-art performance while using less memory and computational resources.