# LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement

arXiv: https://arxiv.org/abs/2504.16053 · HTML: https://mlsanity.com/paper/2504.16053/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Placed in

- State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/)

## Builds on (most cited first, 30 of 31)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [ToolQA: A Dataset for LLM Question Answering with External Tools](https://mlsanity.com/paper/2306.13304/) (2023)
- [Recurrent Memory Transformer](https://mlsanity.com/paper/2207.06881/) (2022)
- [An Empirical Study of Mamba-based Language Models](https://mlsanity.com/paper/2406.07887/) (2024)
- [Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2402.00789/) (2024)
- [Zamba: A Compact 7B SSM Hybrid Model](https://mlsanity.com/paper/2405.16712/) (2024)
- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [Hymba: A Hybrid-head Architecture for Small Language Models](https://mlsanity.com/paper/2411.13676/) (2024)
- [The Hidden Attention of Mamba Models](https://mlsanity.com/paper/2403.01590/) (2024)
- [DeciMamba: Exploring the Length Extrapolation Potential of Mamba](https://mlsanity.com/paper/2406.14528/) (2024)
- [Falcon Mamba: The First Competitive Attention-free 7B Language Model](https://mlsanity.com/paper/2410.05355/) (2024)
- [Resonance RoPE: Improving Context Length Generalization of Large Language Models](https://mlsanity.com/paper/2403.00071/) (2024)
- [The Zamba2 Suite: Technical Report](https://mlsanity.com/paper/2411.15242/) (2024)

## Built on by (most cited first, 30 of 42)

- [TTT3R: 3D Reconstruction as Test-Time Training](https://mlsanity.com/paper/2509.26645/) (2025)
- [Long-Context State-Space Video World Models](https://mlsanity.com/paper/2505.20171/) (2025)
- [Understanding and Improving Length Generalization in Recurrent Models](https://mlsanity.com/paper/2507.02782/) (2025)
- [Context Engineering 2.0: The Context of Context Engineering](https://mlsanity.com/paper/2510.26493/) (2025)
- [Mamba Modulation: On the Length Generalization of Mamba](https://mlsanity.com/paper/2509.19633/) (2025)
- [Comba: Improving Bilinear RNNs with Closed-loop Control](https://mlsanity.com/paper/2506.02475/) (2025)
- [Behavior Knowledge Merge in Reinforced Agentic Models](https://mlsanity.com/paper/2601.13572/) (2026)
- [LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models](https://mlsanity.com/paper/2507.14204/) (2025)
- [Block-Biased Mamba for Long-Range Sequence Processing](https://mlsanity.com/paper/2505.09022/) (2025)
- [Lost in the Noise: How Reasoning Models Fail with Contextual Distractors](https://mlsanity.com/paper/2601.07226/) (2026)
- [Overflow Prevention Enhances Long-Context Recurrent LLMs](https://mlsanity.com/paper/2505.07793/) (2025)
- [Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner](https://mlsanity.com/paper/2506.01301/) (2025)
- [Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data](https://mlsanity.com/paper/2509.17514/) (2025)
- [MemMamba: Rethinking Memory Patterns in State Space Model](https://mlsanity.com/paper/2510.03279/) (2025)
- [MixerCSeg: An Efficient Mixer Architecture for Crack Segmentation via Decoupled Mamba Attention](https://mlsanity.com/paper/2603.01361/) (2026)
- [State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding](https://mlsanity.com/paper/2510.12160/) (2025)
- [Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models](https://mlsanity.com/paper/2602.20981/) (2026)
- [Growing Through Experience: Scaling Episodic Grounding in Language Models](https://mlsanity.com/paper/2506.01312/) (2025)
- [MambaMIL+: Modeling Long-Term Contextual Patterns for Gigapixel Whole Slide Image](https://mlsanity.com/paper/2512.17726/) (2025)
- [MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesis](https://mlsanity.com/paper/2603.12342/) (2026)
- [RS-SSM: Refining Forgotten Specifics in State Space Model for Video Semantic Segmentation](https://mlsanity.com/paper/2603.24295/) (2026)
- [Optimal Decay Spectra for Linear Recurrences](https://mlsanity.com/paper/2604.07658/) (2026)
- [Mamba with Hierarchical Memory: Solving Representation Bottleneck in Long Sequence Modeling](https://mlsanity.com/paper/2608.02347/) (2026)
- [On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashing](https://mlsanity.com/paper/2609.07681/) (2026)
- [SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformers](https://mlsanity.com/paper/2509.00935/) (2025)
- [Chemical Language Models for Natural Products: A State-Space Model Approach](https://mlsanity.com/paper/2602.13958/) (2026)
- [DeformTrace: A Deformable State Space Model with Relay Tokens for Temporal Forgery Localization](https://mlsanity.com/paper/2603.04882/) (2026)
- [Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation](https://mlsanity.com/paper/2603.22333/) (2026)
- [Safety, Security, and Cognitive Risks in State-Space Models: A Systematic Threat Analysis with Spectral, Stateful, and Capacity Attacks](https://mlsanity.com/paper/2604.16424/) (2026)
- [Inference Time Context Sparsity: Illusion or Opportunity?](https://mlsanity.com/paper/2605.24168/) (2026)

## Abstract (arXiv)

State space models (SSMs) have emerged as an efficient alternative to Transformer models for language modeling, offering linear computational complexity and constant memory usage as context length increases. However, despite their efficiency in handling long contexts, recent studies have shown that SSMs, such as Mamba models, generally underperform compared to Transformers in long-context understanding tasks. To address this significant shortfall and achieve both efficient and accurate long-context understanding, we propose LongMamba, a training-free technique that significantly enhances the long-context capabilities of Mamba models. LongMamba builds on our discovery that the hidden channels in Mamba can be categorized into local and global channels based on their receptive field lengths, with global channels primarily responsible for long-context capability. These global channels can become the key bottleneck as the input context lengthens. Specifically, when input lengths largely exceed the training sequence length, global channels exhibit limitations in adaptively extend their receptive fields, leading to Mamba's poor long-context performance. The key idea of LongMamba is to mitigate the hidden state memory decay in these global channels by preventing the accumulation of unimportant tokens in their memory. This is achieved by first identifying critical tokens in the global channels and then applying token filtering to accumulate only those critical tokens. Through extensive benchmarking across synthetic and real-world long-context scenarios, LongMamba sets a new standard for Mamba's long-context performance, significantly extending its operational range without requiring additional training. Our code is available at https://github.com/GATECH-EIC/LongMamba.