mlsanityData to 14 Oct 2026

ReMamba: Equip Mamba with Effective Long-Sequence Modeling

arXiv 2408.15496 ↗First posted 28 Aug 2024cs.CLcited by 13

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

While the Mamba architecture demonstrates superior inference efficiency and competitive performance on short-context natural language processing (NLP) tasks, empirical evidence suggests its capacity to comprehend long contexts is limited compared to transformer-based models. In this study, we investigate the long-context efficiency issues of the Mamba models and propose ReMamba, which enhances Mamba's ability to comprehend long contexts. ReMamba incorporates selective compression and adaptation techniques within a two-stage re-forward process, incurring minimal additional inference costs overhead. Experimental results on the LongBench and L-Eval benchmarks demonstrate ReMamba's efficacy, improving over the baselines by 3.2 and 1.6 points, respectively, and attaining performance almost on par with same-size transformer models.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 21 of its 29 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

21 of 29 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mamba20 Jun
  2. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling11 Jun
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  4. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  5. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks6 Feb
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec

Other references · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  5. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  6. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  7. 2023Adapting Language Models to Compress Contextscited by 356
  8. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  9. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  10. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 147
  11. 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrievalcited by 64
  12. 2021DYLE: Dynamic Latent Extraction for Abstractive Long-Input Summarizationcited by 34
3 more
  1. 2024TransformerFAM: Feedback attention is working memorycited by 22
  2. 2024Do Efficient Transformers Really Save Computation?cited by 20
  3. 2024SelfCP: Compressing Over-Limit Prompt via the Frozen Large Language Model Itselfcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  2. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 23
  3. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  4. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Modelcited by 7
  5. 2025DH-Mamba: Exploring Dual-domain Hierarchical State Space Models for MRI Reconstructioncited by 3
  6. 2025Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolationcited by 3
  7. 2025MedVSR: Medical Video Super-Resolution with Cross State-Space Propagationcited by 3
  8. 2024EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignmentcited by 2
  9. 2025LISM: Long-range Integrative State space Models via Input-Latent State Interactionscited by 1
  10. 2025ReIDMamba: Learning Discriminative Features with Visual State Space Model for Person Re-Identificationcited by 0
  11. 2026Safety, Security, and Cognitive Risks in State-Space Models: A Systematic Threat Analysis with Spectral, Stateful, and Capacity Attackscited by 0
  12. 2026SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterizationcited by 0
1 more
  1. 2026SuPER-Mamba: Surprisal Prioritization for Extended-Context Routing in Mambacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement
  2. 2025MemMamba: Rethinking Memory Patterns in State Space Model
  3. 2024An Empirical Study of Mamba-based Language Models
  4. 2026TurboMamba: Faster Inference, Faster Training, and Better Prediction with Lexical Memory
  5. 2024How Effective are State Space Models for Machine Translation?
  6. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
  7. 2026Mamba with Hierarchical Memory: Solving Representation Bottleneck in Long Sequence Modeling
  8. 2024Stuffed Mamba: Oversized States Lead to the Inability to Forget