mlsanityData to 7 Oct 2026

Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

arXiv 2401.09417 ↗First posted 17 Jan 2024cs.CV cs.LGcited by 1,051

On the map: State-space sequence models (S4, Mamba) → Vision Mamba backbones

Abstract (arXiv)

Recently the state space models (SSMs) with efficient hardware-aware designs, i.e., the Mamba deep learning model, have shown great potential for long sequence modeling. Meanwhile building efficient and generic vision backbones purely upon SSMs is an appealing direction. However, representing visual data is challenging for SSMs due to the position-sensitivity of visual data and the requirement of global context for visual understanding. In this paper, we show that the reliance on self-attention for visual representation learning is not necessary and propose a new generic vision backbone with bidirectional Mamba blocks (Vim), which marks the image sequences with position embeddings and compresses the visual representation with bidirectional state space models. On ImageNet classification, COCO object detection, and ADE20k semantic segmentation tasks, Vim achieves higher performance compared to well-established vision transformers like DeiT, while also demonstrating significantly improved computation & memory efficiency. For example, Vim is 2.8× faster than DeiT and saves 86.8% GPU memory when performing batch inference to extract features on images with a resolution of 1248×1248. The results demonstrate that Vim is capable of overcoming the computation & memory constraints on performing Transformer-style understanding for high-resolution images and it has great potential to be the next-generation backbone for vision foundation models. Code is available at https://github.com/hustvl/Vim.

Key facts. Cited by 1,051 papers in the map (data through 7 Oct 2026). Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 67 of its 79 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

67 of 79 references.

References · 67

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2016Densely Connected Convolutional Networkscited by 6,691
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2022A ConvNet for the 2020scited by 3,545
  7. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  12. 2020Reformer: The Efficient Transformercited by 1,658
  13. 2020Rethinking Attention with Performerscited by 1,584
  14. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  15. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  16. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  17. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  18. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  19. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  20. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  21. 2020Designing Network Design Spacescited by 954
  22. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  23. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  24. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  25. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  26. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  27. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  28. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 595
  29. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  30. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  31. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  32. 2021MetaFormer Is Actually What You Need for Visioncited by 523
  33. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  34. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  35. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  36. 2019Cascade R-CNN: High Quality Object Detection and Instance Segmentationcited by 428
  37. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  38. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 430; 2025: 380; 2026: 241.

1,051 papers.

Sort:

By area:

  1. 2024VMamba: Visual State Space Modelcited by 955
  2. 2024YOLOv9: Learning What You Want to Learn Using Programmable Gradient Informationcited by 422
  3. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 349
  4. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  5. 2025YOLOv12: Attention-Centric Real-Time Object Detectorscited by 300
  6. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  7. 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 286
  8. 2024SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentationcited by 266
  9. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
  10. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  11. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  12. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
38 more
  1. 2024Swin-UMamba: Mamba-based UNet with ImageNet-based pretrainingcited by 179
  2. 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generationcited by 176
  3. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  4. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mambacited by 146
  5. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 142
  6. 2024PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognitioncited by 141
  7. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  8. 2024Graph-Mamba: Towards Long-Range Graph Sequence Modeling with Selective State Spacescited by 132
  9. 2024ChangeMamba: Remote Sensing Change Detection With Spatiotemporal State Space Modelcited by 118
  10. 2024MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detectioncited by 113
  11. 2024Motion Mamba: Efficient and Long Sequence Motion Generationcited by 111
  12. 2024Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splatscited by 111
  13. 2024Is Mamba Effective for Time Series Forecasting?cited by 110
  14. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  15. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 106
  16. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 101
  17. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  18. 2024Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architecturescited by 97
  19. 2024RSMamba: Remote Sensing Image Classification with State Space Modelcited by 95
  20. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  21. 2024VmambaIR: Visual State Space Model for Image Restorationcited by 90
  22. 2024The Hidden Attention of Mamba Modelscited by 88
  23. 2024MambaMIL: Enhancing Long Sequence Modeling with Sequence Reordering in Computational Pathologycited by 87
  24. 2024DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mambacited by 87
  25. 2024MedMamba: Vision Mamba for Medical Image Classificationcited by 86
  26. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85
  27. 2024Mamba-ND: Selective State Space Modeling for Multi-Dimensional Datacited by 84
  28. 2024Graph Mamba: Towards Learning on Graphs with State Space Modelscited by 84
  29. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  30. 2024RS-Mamba for Large Remote Sensing Image Dense Predictioncited by 80
  31. 2024Test-Time Model Adaptation with Only Forward Passescited by 74
  32. 2024Pan-Mamba: Effective pan-sharpening with State Space Modelcited by 73
  33. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesiscited by 72
  34. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 71
  35. 2024Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentationcited by 71
  36. 2024Fusion-Mamba for Cross-modality Object Detectioncited by 67
  37. 2024Point Cloud Mamba: Point Cloud Learning via State Space Modelcited by 66
  38. 2024Voxel Mamba: Group-Free State Space Models for Point Cloud based 3D Object Detectioncited by 66

50 most cited of 1,051 All 1,051 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024V2M: Visual 2-Dimensional Mamba for Image Representation Learning
  2. 2024Visual Mamba: A Survey and New Outlooks
  3. 2026Mask-aware inference with State-Space Models
  4. 2024A Novel State Space Model with Local Enhancement and State Sharing for Image Fusion
  5. 2025MambaNeXt-YOLO: A Hybrid State Space Model for Real-time Object Detection
  6. 2024Computation-Efficient Era: A Comprehensive Survey of State Space Models in Medical Image Analysis
  7. 2024SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series
  8. 2024MSVM-UNet: Multi-Scale Vision Mamba UNet for Medical Image Segmentation