# MemMamba: Rethinking Memory Patterns in State Space Model

arXiv: https://arxiv.org/abs/2510.03279 · HTML: https://mlsanity.com/paper/2510.03279/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 28 Sep 2025.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [An Empirical Study of Mamba-based Language Models](https://mlsanity.com/paper/2406.07887/) (2024)
- [Mamba in Speech: Towards an Alternative to Self-Attention](https://mlsanity.com/paper/2405.12609/) (2024)
- [DeciMamba: Exploring the Length Extrapolation Potential of Mamba](https://mlsanity.com/paper/2406.14528/) (2024)
- [Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length](https://mlsanity.com/paper/2404.08801/) (2024)
- [LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement](https://mlsanity.com/paper/2504.16053/) (2025)
- [Bi-Mamba+: Bidirectional Mamba for Time Series Forecasting](https://mlsanity.com/paper/2404.15772/) (2024)
- [MV-GMN: State Space Model for Multi-View Action Recognition](https://mlsanity.com/paper/2501.13829/) (2025)
- [MOL-Mamba: Enhancing Molecular Representation with Structural & Electronic Insights](https://mlsanity.com/paper/2412.16483/) (2024)
- [MSGM: A Multi-Scale Spatiotemporal Graph Mamba for EEG Emotion Recognition](https://mlsanity.com/paper/2507.15914/) (2025)
- [FastMamba: A High-Speed and Efficient Mamba Accelerator on FPGA with Accurate Quantization](https://mlsanity.com/paper/2505.18975/) (2025)
- [Event USKT : U-State Space Model in Knowledge Transfer for Event Cameras](https://mlsanity.com/paper/2411.15276/) (2024)
- [Hypergraph Mamba for Efficient Whole Slide Image Understanding](https://mlsanity.com/paper/2505.17457/) (2025)

## Built on by (most cited first, 3 of 3)

- [SurgicalMamba: Dual-Path SSD with State Regramming for Online Surgical Phase Recognition](https://mlsanity.com/paper/2605.14889/) (2026)
- [On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models](https://mlsanity.com/paper/2609.16540/) (2026)
- [SchemaMem: Schema-Indexed Recurrent Memory for Delayed State Retrieval](https://mlsanity.com/paper/2609.33436/) (2026)

## Abstract (arXiv)

With the explosive growth of data, long-sequence modeling has become increasingly important in tasks such as natural language processing and bioinformatics. However, existing methods face inherent trade-offs between efficiency and memory. Recurrent neural networks suffer from gradient vanishing and explosion, making them hard to scale. Transformers can model global dependencies but are constrained by quadratic complexity. Recently, selective state-space models such as Mamba have demonstrated high efficiency with O(n) time and O(1) recurrent inference, yet their long-range memory decays exponentially. In this work, we conduct mathematical derivations and information-theoretic analysis to systematically uncover the memory decay mechanism of Mamba, answering a fundamental question: what is the nature of Mamba's long-range memory and how does it retain information? To quantify key information loss, we further introduce horizontal-vertical memory fidelity metrics that capture degradation both within and across layers. Inspired by how humans distill and retain salient information when reading long documents, we propose MemMamba, a novel architectural framework that integrates state summarization mechanism together with cross-layer and cross-token attention, which alleviates long-range forgetting while preserving linear complexity. MemMamba achieves significant improvements over existing Mamba variants and Transformers on long-sequence benchmarks such as PG19 and Passkey Retrieval, while delivering a 48% speedup in inference efficiency. Both theoretical analysis and empirical results demonstrate that MemMamba achieves a breakthrough in the complexity-memory trade-off, offering a new paradigm for ultra-long sequence modeling.