mlsanityData to 7 Oct 2026

The Hidden Attention of Mamba Models

arXiv 2403.01590 ↗First posted 3 Mar 2024cs.LGcited by 88

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

The Mamba layer offers an efficient selective state space model (SSM) that is highly effective in modeling multiple domains, including NLP, long-range sequence processing, and computer vision. Selective SSMs are viewed as dual models, in which one trains in parallel on the entire sequence via an IO-aware parallel scan, and deploys in an autoregressive manner. We add a third view and show that such models can be viewed as attention-driven models. This new perspective enables us to empirically and theoretically compare the underlying mechanisms to that of the self-attention layers in transformers and allows us to peer inside the inner workings of the Mamba model with explainability methods. Our code is publicly available.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 76 of its 84 references that are in the map. First posted on arXiv 3 Mar 2024.

Builds on

76 of 84 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data8 Feb
  2. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks6 Feb
  3. 2024Is Mamba Capable of In-Context Learning?5 Feb
  4. 2024BlackMamba: Mixture of Experts for State-Space Models1 Feb
  5. 2024MambaByte: Token-free Selective State Space Model24 Jan
  6. 2024MambaTab: A Plug-and-Play Model for Learning Tabular Data16 Jan
  7. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts8 Jan
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  9. 2023Structured State Space Models for In-Context Reinforcement Learning7 Mar
  10. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  11. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  12. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  12. 2024VMamba: Visual State Space Modelcited by 955
38 more
  1. 2019Attention is not Explanationcited by 916
  2. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  3. 2020Quantifying Attention Flow in Transformerscited by 759
  4. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  5. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  6. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  7. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  8. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  9. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  10. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  11. 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 286
  12. 2024SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentationcited by 266
  13. 2022Long Range Language Modeling via Gated State Spacescited by 265
  14. 2022Transformer Quality in Linear Timecited by 189
  15. 2024Swin-UMamba: Mamba-based UNet with ImageNet-based pretrainingcited by 179
  16. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  17. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  18. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  19. 2024Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentationcited by 133
  20. 2024Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spacescited by 132
  21. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  22. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
  23. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  24. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  25. 2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
  26. 2022Global Context Vision Transformerscited by 98
  27. 2017Explaining Recurrent Neural Network Predictions in Sentiment Analysiscited by 87
  28. 2024Graph Mamba: Towards Learning on Graphs with State Space Modelscited by 84
  29. 2023Diffusion Models Without Attentioncited by 75
  30. 2023RMT: Retentive Networks Meet Vision Transformerscited by 68
  31. 2023Token Contrast for Weakly-Supervised Semantic Segmentationcited by 64
  32. 2024Vivim: a Video Vision Mamba for Medical Video Segmentationcited by 51
  33. 2019Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networkscited by 49
  34. 2021Vision Transformers with Patch Diversificationcited by 45
  35. 2024nnMamba: 3D Biomedical Image Segmentation, Classification and Landmark Detection with State Space Modelcited by 45
  36. 2023Replacing softmax with ReLU in Vision Transformerscited by 43
  37. 2022Optimizing Relevance Maps of Vision Transformers Improves Robustnesscited by 41
  38. 2023Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionalscited by 31

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 44; 2025: 31; 2026: 13.

88 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  2. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  3. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  4. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 101
  5. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  6. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  7. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  8. 2024TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecastingcited by 63
  9. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  10. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  11. 2024Visual Mamba: A Survey and New Outlookscited by 41
  12. 2024MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selectioncited by 40
38 more
  1. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 36
  2. 2024Locating and Editing Factual Associations in Mambacited by 35
  3. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 31
  4. 2024Towards Universality: Studying Mechanistic Similarity Across Language Model Architecturescited by 31
  5. 2024Exploring Token Pruning in Vision State Space Modelscited by 27
  6. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  7. 2024VFIMamba: Video Frame Interpolation with State Space Modelscited by 22
  8. 2024State Space Models as Foundation Models: A Control Theoretic Overviewcited by 18
  9. 2024Mamba in Vision: A Comprehensive Survey of Techniques and Applicationscited by 18
  10. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  11. 2024Mimetic Initialization Helps State Space Models Learn to Recallcited by 15
  12. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  13. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  14. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  15. 2024TSCMamba: Mamba Meets Multi-View Learning for Time Series Classificationcited by 10
  16. 2024Provable Benefits of Complex Parameterizations for Structured State Space Modelscited by 9
  17. 2024Deep Continuous-Time State-Space Models for Marked Event Sequencescited by 9
  18. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  19. 2025MamBEV: Enabling State Space Models to Learn Birds-Eye-View Representationscited by 8
  20. 2024Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?cited by 7
  21. 2025Revisiting associative recall in modern recurrent modelscited by 7
  22. 2024Mamba4KT:An Efficient and Effective Mamba-based Knowledge Tracing Modelcited by 6
  23. 20241.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Datacited by 6
  24. 2025Dynamic Vision Mambacited by 6
  25. 2025Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainabilitycited by 6
  26. 2024Mamba-ST: State Space Model for Efficient Style Transfercited by 5
  27. 2025Generalization Error Analysis for Selective State-Space Models Through the Lens of Attentioncited by 5
  28. 2025OuroMamba: A Data-Free Quantization Framework for Vision Mambacited by 5
  29. 2024Cross-attention Inspired Selective State Space Models for Target Sound Extractioncited by 4
  30. 2024Lambda-Skip Connections: the architectural component that prevents Rank Collapsecited by 4
  31. 2024START: A Generalized State Space Model with Saliency-Driven Token-Aware Transformationcited by 4
  32. 2025Mamba Knockout for Unraveling Factual Information Flowcited by 4
  33. 2025Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridgecited by 4
  34. 2025TimeViper: A Hybrid Mamba-Transformer Vision-Language Model for Efficient Long Video Understandingcited by 4
  35. 2024UmambaTSF: A U-shaped Multi-Scale Long-Term Time Series Forecasting Method Using Mambacited by 3
  36. 2024NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMscited by 3
  37. 2024A Comparative Study on Dynamic Graph Embedding based on Mamba and Transformerscited by 3
  38. 2025On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approachcited by 3

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Visual Attention Exploration in Vision-Based Mamba Models
  2. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
  3. 2026InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model
  4. 2024An Empirical Study of Mamba-based Language Models
  5. 2025A2Mamba: Attention-augmented State Space Models for Visual Recognition
  6. 2024MambaOut: Do We Really Need Mamba for Vision?
  7. 2024A Survey on Visual Mamba
  8. 2025A Separable Self-attention Inspired by the State Space Model for Computer Vision