mlsanityData to 7 Oct 2026

2Mamba2Furious: Linear in Complexity, Competitive in Accuracy

arXiv 2602.17363 ↗First posted 19 Feb 2026cs.LGcited by 1

On the map: State-space sequence models (S4, Mamba) → Vision Mamba backbones

Abstract (arXiv)

Linear attention transformers have become a strong alternative to softmax attention due to their efficiency. However, linear attention tends to be less expressive and results in reduced accuracy compared to softmax attention. To bridge the accuracy gap between softmax attention and linear attention, we manipulate Mamba-2, a very strong linear attention variant. We first simplify Mamba-2 down to its most fundamental and important components, evaluating which specific choices make it most accurate. From this simplified Mamba variant (Mamba-2S), we improve the A-mask and increase the order of the hidden state, resulting in a method, which we call 2Mamba, that is nearly as accurate as softmax attention, yet much more memory efficient for long context lengths. We also investigate elements to Mamba-2 that help surpass softmax attention accuracy. Code is provided for all our experiments.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 27 of its 32 references that are in the map. First posted on arXiv 19 Feb 2026.

Builds on

27 of 32 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2020Rethinking Attention with Performerscited by 1,584
15 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  6. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  7. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  8. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  9. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  10. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  11. 2018Online normalizer calculation for softmaxcited by 154
  12. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  13. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  14. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  15. 2025On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspectivecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Don't Read Everything: A Curvature-Conditioned Query for Linear Attentioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Demystify Mamba in Vision: A Linear Attention Perspective
  2. 2025RNN as Linear Transformer: A Closer Investigation into Representational Potentials of Visual Mamba Models
  3. 2024The Hidden Attention of Mamba Models
  4. 2024MambaOut: Do We Really Need Mamba for Vision?
  5. 2024Mamba in Speech: Towards an Alternative to Self-Attention
  6. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
  7. 2025MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modeling
  8. 2025SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging