# LocalMamba: Visual State Space Model with Windowed Selective Scan

arXiv: https://arxiv.org/abs/2403.09338 · HTML: https://mlsanity.com/paper/2403.09338/

## Key facts

- Cited by 236 papers in the map (data through 14 Oct 2026).
- Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 50 of its 58 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Vision Mamba backbones](https://mlsanity.com/ssm/vision-mamba-backbones/)

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)
- [Twins: Revisiting the Design of Spatial Attention in Vision Transformers](https://mlsanity.com/paper/2104.13840/) (2021)

## Built on by (most cited first, 30 of 236)

- [MambaOut: Do We Really Need Mamba for Vision?](https://mlsanity.com/paper/2405.07992/) (2024)
- [PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition](https://mlsanity.com/paper/2403.17695/) (2024)
- [Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats](https://mlsanity.com/paper/2410.12781/) (2024)
- [MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection](https://mlsanity.com/paper/2404.06564/) (2024)
- [Demystify Mamba in Vision: A Linear Attention Perspective](https://mlsanity.com/paper/2405.16605/) (2024)
- [Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges](https://mlsanity.com/paper/2404.16112/) (2024)
- [Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model](https://mlsanity.com/paper/2405.14174/) (2024)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [FreqMamba: Viewing Mamba from a Frequency Perspective for Image Deraining](https://mlsanity.com/paper/2404.09476/) (2024)
- [RainMamba: Enhanced Locality Learning with State Space Models for Video Deraining](https://mlsanity.com/paper/2407.21773/) (2024)
- [Vision Mamba: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2405.04404/) (2024)
- [A Survey on Visual Mamba](https://mlsanity.com/paper/2404.15956/) (2024)
- [MambaLLIE: Implicit Retinex-Aware Low Light Enhancement with Global-then-Local State Space](https://mlsanity.com/paper/2405.16105/) (2024)
- [CDMamba: Incorporating Local Clues into Mamba for Remote Sensing Image Binary Change Detection](https://mlsanity.com/paper/2406.04207/) (2024)
- [MaIR: A Locality- and Continuity-Preserving Mamba for Image Restoration](https://mlsanity.com/paper/2412.20066/) (2024)
- [Mamba-R: Vision Mamba ALSO Needs Registers](https://mlsanity.com/paper/2405.14858/) (2024)
- [Spatial-Mamba: Effective Visual State Space Models via Structure-aware State Fusion](https://mlsanity.com/paper/2410.15091/) (2024)
- [MambaMixer: Efficient Selective State Space Models with Dual Token and Channel Selection](https://mlsanity.com/paper/2403.19888/) (2024)
- [A Survey of Mamba](https://mlsanity.com/paper/2408.01129/) (2024)
- [2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification](https://mlsanity.com/paper/2412.00678/) (2024)
- [VSSD: Vision Mamba with Non-Causal State Space Duality](https://mlsanity.com/paper/2407.18559/) (2024)
- [MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods](https://mlsanity.com/paper/2501.13484/) (2025)
- [Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More](https://mlsanity.com/paper/2502.03738/) (2025)
- [MVGamba: Unify 3D Content Generation as State Space Sequence Modeling](https://mlsanity.com/paper/2406.06367/) (2024)
- [Autoregressive Pretraining with Mamba in Vision](https://mlsanity.com/paper/2406.07537/) (2024)
- [MambaIC: State Space Models for High-Performance Learned Image Compression](https://mlsanity.com/paper/2503.12461/) (2025)
- [Mamba in Vision: A Comprehensive Survey of Techniques and Applications](https://mlsanity.com/paper/2410.03105/) (2024)
- [QuadMamba: Learning Quadtree-based Selective Scan for Visual State Space Model](https://mlsanity.com/paper/2410.06806/) (2024)
- [COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection](https://mlsanity.com/paper/2412.18076/) (2024)
- [DefMamba: Deformable Visual State Space Model](https://mlsanity.com/paper/2504.05794/) (2025)

## Abstract (arXiv)

Recent advancements in state space models, notably Mamba, have demonstrated significant progress in modeling long sequences for tasks like language understanding. Yet, their application in vision tasks has not markedly surpassed the performance of traditional Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). This paper posits that the key to enhancing Vision Mamba (ViM) lies in optimizing scan directions for sequence modeling. Traditional ViM approaches, which flatten spatial tokens, overlook the preservation of local 2D dependencies, thereby elongating the distance between adjacent tokens. We introduce a novel local scanning strategy that divides images into distinct windows, effectively capturing local dependencies while maintaining a global perspective. Additionally, acknowledging the varying preferences for scan patterns across different network layers, we propose a dynamic method to independently search for the optimal scan choices for each layer, substantially improving performance. Extensive experiments across both plain and hierarchical models underscore our approach's superiority in effectively capturing image representations. For example, our model significantly outperforms Vim-Ti by 3.1% on ImageNet with the same 1.5G FLOPs. Code is available at: https://github.com/hunto/LocalMamba.