# ReMamba: Equip Mamba with Effective Long-Sequence Modeling

arXiv: https://arxiv.org/abs/2408.15496 · HTML: https://mlsanity.com/paper/2408.15496/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 28 Aug 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Jamba: A Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2403.19887/) (2024)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Orca: Progressive Learning from Complex Explanation Traces of GPT-4](https://mlsanity.com/paper/2306.02707/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [L-Eval: Instituting Standardized Evaluation for Long Context Language Models](https://mlsanity.com/paper/2307.11088/) (2023)
- [Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling](https://mlsanity.com/paper/2406.07522/) (2024)
- [PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference](https://mlsanity.com/paper/2405.12532/) (2024)
- [Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks](https://mlsanity.com/paper/2402.04248/) (2024)
- [RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval](https://mlsanity.com/paper/2402.18510/) (2024)
- [DeciMamba: Exploring the Length Extrapolation Potential of Mamba](https://mlsanity.com/paper/2406.14528/) (2024)
- [DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarization](https://mlsanity.com/paper/2110.08168/) (2021)
- [TransformerFAM: Feedback attention is working memory](https://mlsanity.com/paper/2404.09173/) (2024)
- [Do Efficient Transformers Really Save Computation?](https://mlsanity.com/paper/2402.13934/) (2024)
- [SelfCP: Compressing Over-Limit Prompt via the Frozen Large Language Model Itself](https://mlsanity.com/paper/2405.17052/) (2024)

## Built on by (most cited first, 13 of 13)

- [Why Does the Effective Context Length of LLMs Fall Short?](https://mlsanity.com/paper/2410.18745/) (2024)
- [Understanding and Improving Length Generalization in Recurrent Models](https://mlsanity.com/paper/2507.02782/) (2025)
- [Reasoning Beyond Limits: Advances and Open Problems for LLMs](https://mlsanity.com/paper/2503.22732/) (2025)
- [TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model](https://mlsanity.com/paper/2503.24067/) (2025)
- [DH-Mamba: Exploring Dual-domain Hierarchical State Space Models for MRI Reconstruction](https://mlsanity.com/paper/2501.08163/) (2025)
- [Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation](https://mlsanity.com/paper/2505.03320/) (2025)
- [MedVSR: Medical Video Super-Resolution with Cross State-Space Propagation](https://mlsanity.com/paper/2509.21265/) (2025)
- [EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment](https://mlsanity.com/paper/2410.05938/) (2024)
- [LISM: Long-range Integrative State space Models via Input-Latent State Interactions](https://mlsanity.com/paper/2509.04226/) (2025)
- [ReIDMamba: Learning Discriminative Features with Visual State Space Model for Person Re-Identification](https://mlsanity.com/paper/2511.07948/) (2025)
- [Safety, Security, and Cognitive Risks in State-Space Models: A Systematic Threat Analysis with Spectral, Stateful, and Capacity Attacks](https://mlsanity.com/paper/2604.16424/) (2026)
- [SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization](https://mlsanity.com/paper/2609.14320/) (2026)
- [SuPER-Mamba: Surprisal Prioritization for Extended-Context Routing in Mamba](https://mlsanity.com/paper/2610.16636/) (2026)

## Abstract (arXiv)

While the Mamba architecture demonstrates superior inference efficiency and competitive performance on short-context natural language processing (NLP) tasks, empirical evidence suggests its capacity to comprehend long contexts is limited compared to transformer-based models. In this study, we investigate the long-context efficiency issues of the Mamba models and propose ReMamba, which enhances Mamba's ability to comprehend long contexts. ReMamba incorporates selective compression and adaptation techniques within a two-stage re-forward process, incurring minimal additional inference costs overhead. Experimental results on the LongBench and L-Eval benchmarks demonstrate ReMamba's efficacy, improving over the baselines by 3.2 and 1.6 points, respectively, and attaining performance almost on par with same-size transformer models.