mlsanityData to 14 Oct 2026

Block-Biased Mamba for Long-Range Sequence Processing

arXiv 2505.09022 ↗First posted 13 May 2025cs.LG cs.AI stat.MLcited by 8

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Mamba extends earlier state space models (SSMs) by introducing input-dependent dynamics, and has demonstrated strong empirical performance across a range of domains, including language modeling, computer vision, and foundation models. However, a surprising weakness remains: despite being built on architectures designed for long-range dependencies, Mamba performs poorly on long-range sequential tasks. Understanding and addressing this gap is important for improving Mamba's universality and versatility. In this work, we analyze Mamba's limitations through three perspectives: expressiveness, inductive bias, and training stability. Our theoretical results show how Mamba falls short in each of these aspects compared to earlier SSMs such as S4D. To address these issues, we propose B2S6, a simple extension of Mamba's S6 unit that combines block-wise selective dynamics with a channel-specific bias. We prove that these changes equip the model with a better-suited inductive bias and improve its expressiveness and stability. Empirically, B2S6 outperforms S4 and S4D on Long-Range Arena (LRA) tasks while maintaining Mamba's performance on language modeling benchmarks.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 91 of its 117 references that are in the map. First posted on arXiv 13 May 2025.

Builds on

91 of 117 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement22 Apr
  2. 2025Generalization Error Analysis for Selective State-Space Models Through the Lens of Attention3 Feb
  3. 2025SeRpEnt: Selective Resampling for Expressive State Space Models20 Jan
  4. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues19 Nov
  5. 2024Provable Benefits of Complex Parameterizations for Structured State Space Models17 Oct
  6. 2024SIGMA: Selective Gated Mamba for Sequential Recommendation21 Aug
  7. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers13 Jul
  8. 2024An Empirical Study of Mamba-based Language Models12 Jun
  9. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  10. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges24 Apr
  11. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  12. 2024Theoretical Foundations of Deep Selective State-Space Models29 Feb
4 more
  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  2. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  3. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  4. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 75

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  7. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 2,018
  10. 2020Reformer: The Efficient Transformercited by 1,668
  11. 2020Rethinking Attention with Performerscited by 1,601
  12. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
38 more
  1. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  2. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  3. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
  4. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  5. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  6. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  7. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  8. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  9. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  10. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  11. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  12. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
  13. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  14. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  15. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  16. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  17. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
  18. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  19. 2014The Loss Surfaces of Multilayer Networkscited by 280
  20. 2016Benefits of depth in neural networkscited by 250
  21. 2021Deep learning: a statistical viewpointcited by 240
  22. 2019Universal Approximation with Deep Narrow Networkscited by 200
  23. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  24. 2018Universality of Deep Convolutional Neural Networkscited by 176
  25. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  26. 2022Liquid Structural State-Space Modelscited by 136
  27. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 135
  28. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 120
  29. 2024Is Mamba Effective for Time Series Forecasting?cited by 114
  30. 2019AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networkscited by 112
  31. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 106
  32. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  33. 2020Lipschitz Recurrent Neural Networkscited by 89
  34. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  35. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  36. 2024A Survey on Visual Mambacited by 50
  37. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 39
  38. 2024Oscillatory State-Space Modelscited by 39

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 7.

8 papers.

  1. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 89
  2. 2026Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Trackingcited by 8
  3. 2026A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Modelscited by 3
  4. 2025LISM: Long-range Integrative State space Models via Input-Latent State Interactionscited by 1
  5. 2026Product Interaction: An Algebraic Formalism for Deep Learning Architecturescited by 0
  6. 2026Continuity Laws for Sequential Modelscited by 0
  7. 2026Inference Time Context Sparsity: Illusion or Opportunity?cited by 0
  8. 2026The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
  2. 2024Bi-Mamba: Towards Accurate 1-Bit State Space Models
  3. 2024The Hidden Attention of Mamba Models
  4. 2025Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
  5. 2025Controllability Analysis of State Space-based Language Model
  6. 2024A Survey of Mamba
  7. 2025On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach
  8. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba