mlsanityData to 7 Oct 2026

DeciMamba: Exploring the Length Extrapolation Potential of Mamba

arXiv 2406.14528 ↗First posted 20 Jun 2024cs.LG cs.AIcited by 45

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Long-range sequence processing poses a significant challenge for Transformers due to their quadratic complexity in input length. A promising alternative is Mamba, which demonstrates high performance and achieves Transformer-level capabilities while requiring substantially fewer computational resources. In this paper we explore the length-generalization capabilities of Mamba, which we find to be relatively limited. Through a series of visualizations and analyses we identify that the limitations arise from a restricted effective receptive field, dictated by the sequence length used during training. To address this constraint, we introduce DeciMamba, a context-extension method specifically designed for Mamba. This mechanism, built on top of a hidden filtering mechanism embedded within the S6 layer, enables the trained model to extrapolate well even without additional training. Empirical experiments over real-world long-range NLP tasks show that DeciMamba can extrapolate to context lengths that are significantly longer than the ones seen during training, while enjoying faster inference.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 59 of its 65 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

59 of 65 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024MambaLRP: Explaining Selective State Space Sequence Models11 Jun
  2. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  3. 2024The Hidden Attention of Mamba Models3 Mar
  4. 2024MambaByte: Token-free Selective State Space Model24 Jan
  5. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts8 Jan
  6. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  8. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  9. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2020Rethinking Attention with Performerscited by 1,584
38 more
  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  4. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  5. 2024VMamba: Visual State Space Modelcited by 955
  6. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  7. 2020Efficient Transformers: A Surveycited by 784
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  9. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  10. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  12. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  13. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  14. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  15. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  16. 2023Adapting Language Models to Compress Contextscited by 350
  17. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  18. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  19. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  20. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  21. 2022Long Range Language Modeling via Gated State Spacescited by 265
  22. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  23. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  24. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  25. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  26. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  27. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  28. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  29. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 185
  30. 2022A Length-Extrapolatable Transformercited by 123
  31. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  32. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  33. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
  34. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  35. 2024Contextual Position Encoding: Learning to Count What's Importantcited by 53
  36. 2023Extending Context Window of Large Language Models via Semantic Compressioncited by 45
  37. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 39
  38. 2024SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Modelcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 23; 2026: 15.

45 papers.

Sort:

By area:

  1. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  2. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  3. 2025Long-Context State-Space Video World Modelscited by 59
  4. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 36
  5. 2025GENERator: A Long-Context Generative Genomic Foundation Modelcited by 20
  6. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 19
  7. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  8. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  9. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  10. 2024ReMamba: Equip Mamba with Effective Long-Sequence Modelingcited by 12
  11. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 12
  12. 2024Provable Benefits of Complex Parameterizations for Structured State Space Modelscited by 9
33 more
  1. 2024Stuffed Mamba: Oversized States Lead to the Inability to Forgetcited by 7
  2. 2025Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modelingcited by 7
  3. 2024Teaching VLMs to Localize Specific Objects from In-context Examplescited by 6
  4. 2025Overflow Prevention Enhances Long-Context Recurrent LLMscited by 5
  5. 2025Mamba Knockout for Unraveling Factual Information Flowcited by 4
  6. 2025Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modelingcited by 4
  7. 2025On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attentioncited by 3
  8. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic datacited by 3
  9. 2025MemMamba: Rethinking Memory Patterns in State Space Modelcited by 3
  10. 2025To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Modelscited by 3
  11. 2026MixerCSeg: An Efficient Mixer Architecture for Crack Segmentation via Decoupled Mamba Attentioncited by 3
  12. 2024Revealing and Mitigating the Local Pattern Shortcuts of Mambacited by 2
  13. 2025Differential Mambacited by 2
  14. 2025MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modelingcited by 2
  15. 2026What Attention Recalls and Recurrence Controls in Hybrid Language Modelscited by 2
  16. 2025MambaMIL+: Modeling Long-Term Contextual Patterns for Gigapixel Whole Slide Imagecited by 1
  17. 2026CLIMP: Contrastive Language-Image Mamba Pretrainingcited by 1
  18. 2026Optimal Decay Spectra for Linear Recurrencescited by 1
  19. 2026On the Recall Scaling Laws in Mamba: A Theoretical and Mechanistic Study via Hashingcited by 1
  20. 2026Anatomy of Associative Recall in Fixed-State Recurrences: A Matched-State Decomposition, an Interference Wall, and a Curriculum That Breaks Itcited by 1
  21. 2025Technical Debt in In-Context Learning: Diminishing Efficiency in Long Contextcited by 0
  22. 2025A Novel Mamba-based Sequential Recommendation Methodcited by 0
  23. 2025Generalizable Trajectory Prediction via Inverse Reinforcement Learning with Mamba-Graph Architecturecited by 0
  24. 2025SCOUT: Toward Sub-Quadratic Attention via Segment Compression for Optimized Utility in Transformerscited by 0
  25. 2026Probing Length Generalization in Mamba via Image Reconstructioncited by 0
  26. 2026Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulationcited by 0
  27. 2026Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matchingcited by 0
  28. 2026Safety, Security, and Cognitive Risks in State-Space Models: A Systematic Threat Analysis with Spectral, Stateful, and Capacity Attackscited by 0
  29. 2026Mamba with Hierarchical Memory: Solving Representation Bottleneck in Long Sequence Modelingcited by 0
  30. 2026SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Modelscited by 0
  31. 2026SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterizationcited by 0
  32. 2026On the Importance of Gating: Memorization vs. In-Context Learning in State Space Modelscited by 0
  33. 2026A Self-Pruning Transformer: Extreme KV-Cache Compression with Universal Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
  2. 2024PackMamba: Efficient Processing of Variable-Length Sequences in Mamba training
  3. 2024An Empirical Study of Mamba-based Language Models
  4. 2024A Survey of Mamba
  5. 2023Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation
  6. 2025DYNAMAX: Dynamic computing for Transformers and Mamba based architectures
  7. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMs
  8. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving