mlsanityData to 7 Oct 2026

Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mamba

arXiv 2406.16722 ↗First posted 24 Jun 2024cs.CLcited by 1

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Transformer, a deep neural network architecture, has long dominated the field of natural language processing and beyond. Nevertheless, the recent introduction of Mamba challenges its supremacy, sparks considerable interest among researchers, and gives rise to a series of Mamba-based models that have exhibited notable potential. This survey paper orchestrates a comprehensive discussion, diving into essential research dimensions, covering: (i) the functioning of the Mamba mechanism and its foundation on the principles of structured state space models; (ii) the proposed improvements and the integration of Mamba with various networks, exploring its potential as a substitute for Transformers; (iii) the combination of Transformers and Mamba to compensate for each other's shortcomings. We have also made efforts to interpret Mamba and Transformer in the framework of kernel functions, allowing for a comparison of their mathematical nature within a unified context. Our paper encompasses the vast majority of improvements related to Mamba to date.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 123 of its 147 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

123 of 147 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling11 Jun
  2. 2024EchoMamba4Rec: Harmonizing Bidirectional State Space Models with Spectral Filtering for Advanced Sequential Recommendation4 Jun
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  4. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges24 Apr
  5. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  6. 2024Simple linear attention language models balance the recall-throughput tradeoff28 Feb
  7. 2024Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data8 Feb
  8. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks6 Feb
  9. 2024BlackMamba: Mixture of Experts for State-Space Models1 Feb
  10. 2024MambaByte: Token-free Selective State Space Model24 Jan
  11. 2024MambaTab: A Plug-and-Play Model for Learning Tabular Data16 Jan
  12. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts8 Jan
3 more
  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  2. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projections24 Jun
  3. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 108

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  10. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  11. 2024VMamba: Visual State Space Modelcited by 955
  12. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
38 more
  1. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  2. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  3. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  4. 2019Graph Transformer Networkscited by 520
  5. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  6. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  7. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  8. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 349
  9. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  10. 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 286
  11. 2024SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentationcited by 266
  12. 2022Long Range Graph Benchmarkcited by 258
  13. 2023A Survey on Oversmoothing in Graph Neural Networkscited by 246
  14. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
  15. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  16. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  17. 2024Swin-UMamba: Mamba-based UNet with ImageNet-based pretrainingcited by 179
  18. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  19. 2023On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topologycited by 163
  20. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mambacited by 146
  21. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 142
  22. 2024Mamba-UNet: UNet-Like Pure Visual Mamba for Medical Image Segmentationcited by 133
  23. 2024Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spacescited by 132
  24. 2024ChangeMamba: Remote Sensing Change Detection With Spatiotemporal State Space Modelcited by 118
  25. 2024Motion Mamba: Efficient and Long Sequence Motion Generationcited by 111
  26. 2024Is Mamba Effective for Time Series Forecasting?cited by 110
  27. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  28. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  29. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  30. 2024RSMamba: Remote Sensing Image Classification with State Space Modelcited by 95
  31. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  32. 2024MedMamba: Vision Mamba for Medical Image Classificationcited by 86
  33. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85
  34. 2024Graph Mamba: Towards Learning on Graphs with State Space Modelscited by 84
  35. 2024Pan-Mamba: Effective pan-sharpening with State Space Modelcited by 73
  36. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesiscited by 72
  37. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 71
  38. 2024Point Cloud Mamba: Point Cloud Learning via State Space Modelcited by 66

50 most cited of 108 All 108 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Survey on Mamba Architecture for Vision Applicationscited by 8

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey of Mamba
  2. 2024An Empirical Study of Mamba-based Language Models
  3. 2024Sparse Mamba: Introducing Controllability, Observability, And Stability To Structural State Space Models
  4. 2025Differential Mamba
  5. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
  6. 2025Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data
  7. 2025DYNAMAX: Dynamic computing for Transformers and Mamba based architectures
  8. 2024Speech-Mamba: Long-Context Speech Recognition with Selective State Spaces Models