# Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model

arXiv: https://arxiv.org/abs/2405.14174 · HTML: https://mlsanity.com/paper/2405.14174/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 62 references that are in the map.
- First posted on arXiv 23 May 2024.

## Builds on (most cited first, 30 of 55)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Transformer in Transformer](https://mlsanity.com/paper/2103.00112/) (2021)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)

## Built on by (most cited first, 30 of 62)

- [A Survey of Mamba](https://mlsanity.com/paper/2408.01129/) (2024)
- [2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification](https://mlsanity.com/paper/2412.00678/) (2024)
- [VSSD: Vision Mamba with Non-Causal State Space Duality](https://mlsanity.com/paper/2407.18559/) (2024)
- [SMamba: Sparse Mamba for Event-based Object Detection](https://mlsanity.com/paper/2501.11971/) (2025)
- [Mamba in Vision: A Comprehensive Survey of Techniques and Applications](https://mlsanity.com/paper/2410.03105/) (2024)
- [DefMamba: Deformable Visual State Space Model](https://mlsanity.com/paper/2504.05794/) (2025)
- [EfficientViM: Efficient Vision Mamba with Hidden State Mixer based State Space Duality](https://mlsanity.com/paper/2411.15241/) (2024)
- [MobileMamba: Lightweight Multi-Receptive Visual Mamba Network](https://mlsanity.com/paper/2411.15941/) (2024)
- [Rectifying Magnitude Neglect in Linear Attention](https://mlsanity.com/paper/2507.00698/) (2025)
- [TinyViM: Frequency Decoupling for Tiny Hybrid Vision Mamba](https://mlsanity.com/paper/2411.17473/) (2024)
- [MambaCSR: Dual-Interleaved Scanning for Compressed Image Super-Resolution With SSMs](https://mlsanity.com/paper/2408.11758/) (2024)
- [LoG-VMamba: Local-Global Vision Mamba for Medical Image Segmentation](https://mlsanity.com/paper/2408.14415/) (2024)
- [MambaVLT: Time-Evolving Multimodal State Space Model for Vision-Language Tracking](https://mlsanity.com/paper/2411.15459/) (2024)
- [Physics-Driven Autoregressive State Space Models for Medical Image Reconstruction](https://mlsanity.com/paper/2412.09331/) (2024)
- [Shuffle Mamba: State Space Models with Random Shuffle for Multi-Modal Image Fusion](https://mlsanity.com/paper/2409.01728/) (2024)
- [MHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Head](https://mlsanity.com/paper/2601.07832/) (2026)
- [Dynamic Vision Mamba](https://mlsanity.com/paper/2504.04787/) (2025)
- [Vision Mamba in Remote Sensing: A Comprehensive Survey of Techniques, Applications and Outlook](https://mlsanity.com/paper/2505.00630/) (2025)
- [EMOv2: Pushing 5M Vision Model Frontier](https://mlsanity.com/paper/2412.06674/) (2024)
- [Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training](https://mlsanity.com/paper/2412.12496/) (2024)
- [MambaGlue: Fast and Robust Local Feature Matching With Mamba](https://mlsanity.com/paper/2502.00462/) (2025)
- [Spectral State Space Model for Rotation-Invariant Visual Representation Learning](https://mlsanity.com/paper/2503.06369/) (2025)
- [An Efficient and Mixed Heterogeneous Model for Image Restoration](https://mlsanity.com/paper/2504.10967/) (2025)
- [SF-Mamba: Rethinking State Space Model for Vision](https://mlsanity.com/paper/2603.16423/) (2026)
- [Mamba2D: A Natively Multi-Dimensional State-Space Model for Vision Tasks](https://mlsanity.com/paper/2412.16146/) (2024)
- [MANTA: Diffusion Mamba for Efficient and Effective Stochastic Long-Term Dense Anticipation](https://mlsanity.com/paper/2501.08837/) (2025)
- [BioMamba: Leveraging Spectro-Temporal Embedding in Bidirectional Mamba for Enhanced Biosignal Classification](https://mlsanity.com/paper/2503.11741/) (2025)
- [HAMSA: Scanning-Free Vision State Space Models via SpectralPulseNet](https://mlsanity.com/paper/2604.14724/) (2026)
- [Mamba-Adaptor: State Space Model Adaptor for Visual Recognition](https://mlsanity.com/paper/2505.12685/) (2025)
- [Deformba: Vision State Space Model with Adaptive State Fusion](https://mlsanity.com/paper/2605.21308/) (2026)

## Abstract (arXiv)

Despite the significant achievements of Vision Transformers (ViTs) in various vision tasks, they are constrained by the quadratic complexity. Recently, State Space Models (SSMs) have garnered widespread attention due to their global receptive field and linear complexity with respect to the input length, demonstrating substantial potential across fields including natural language processing and computer vision. To improve the performance of SSMs in vision tasks, a multi-scan strategy is widely adopted, which leads to significant redundancy of SSMs. For a better trade-off between efficiency and performance, we analyze the underlying reasons behind the success of the multi-scan strategy, where long-range dependency plays an important role. Based on the analysis, we introduce Multi-Scale Vision Mamba (MSVMamba) to preserve the superiority of SSMs in vision tasks with limited parameters. It employs a multi-scale 2D scanning technique on both original and downsampled feature maps, which not only benefits long-range dependency learning but also reduces computational costs. Additionally, we integrate a Convolutional Feed-Forward Network (ConvFFN) to address the lack of channel mixing. Our experiments demonstrate that MSVMamba is highly competitive, with the MSVMamba-Tiny model achieving 82.8% top-1 accuracy on ImageNet, 46.9% box mAP, and 42.2% instance mAP with the Mask R-CNN framework, 1x training schedule on COCO, and 47.6% mIoU with single-scale testing on ADE20K.Code is available at urlhttps://github.com/YuHengsss/MSVMamba.