# VSSD: Vision Mamba with Non-Causal State Space Duality

arXiv: https://arxiv.org/abs/2407.18559 · HTML: https://mlsanity.com/paper/2407.18559/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 67 of its 72 references that are in the map.
- First posted on arXiv 26 Jul 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Vision Mamba backbones](https://mlsanity.com/ssm/vision-mamba-backbones/)

## Builds on (most cited first, 30 of 67)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)

## Built on by (most cited first, 29 of 29)

- [Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiency](https://mlsanity.com/paper/2410.07599/) (2024)
- [A Survey on Mamba Architecture for Vision Applications](https://mlsanity.com/paper/2502.07161/) (2025)
- [Mamba-Sea: A Mamba-based Framework with Global-to-Local Sequence Augmentation for Generalizable Medical Image Segmentation](https://mlsanity.com/paper/2504.17515/) (2025)
- [Vision Mamba in Remote Sensing: A Comprehensive Survey of Techniques, Applications and Outlook](https://mlsanity.com/paper/2505.00630/) (2025)
- [MODEM: A Morton-Order Degradation Estimation Mechanism for Adverse Weather Image Recovery](https://mlsanity.com/paper/2505.17581/) (2025)
- [Mamba2D: A Natively Multi-Dimensional State-Space Model for Vision Tasks](https://mlsanity.com/paper/2412.16146/) (2024)
- [Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models](https://mlsanity.com/paper/2602.20981/) (2026)
- [Deformba: Vision State Space Model with Adaptive State Fusion](https://mlsanity.com/paper/2605.21308/) (2026)
- [DensePercept-NCSSD: Vision Mamba towards Real-time Dense Visual Perception with Non-Causal State Space Duality](https://mlsanity.com/paper/2511.12671/) (2025)
- [CellMamba: Adaptive Mamba for Accurate and Efficient Cell Detection](https://mlsanity.com/paper/2512.21803/) (2025)
- [Partial Ring Scan: Revisiting Scan Order in Vision State Space Models](https://mlsanity.com/paper/2602.04170/) (2026)
- [Vision Non-Causal Trapezoidal Mamba: Eliminating Directional Scanning in Vision SSMs with Second-Order Dynamics](https://mlsanity.com/paper/2607.03589/) (2026)
- [Nd-BiMamba2: A Unified Bidirectional Architecture for Multi-Dimensional Data Processing](https://mlsanity.com/paper/2411.15380/) (2024)
- [Shallow- and Deep-fake Image Manipulation Localization Using Vision Mamba and Guided Graph Neural Network](https://mlsanity.com/paper/2601.02566/) (2026)
- [PanopMamba: Vision State Space Modeling for Nuclei Panoptic Segmentation](https://mlsanity.com/paper/2601.16631/) (2026)
- [DenVisCoM: Dense Vision Correspondence Mamba for Efficient and Real-time Optical Flow and Stereo Estimation](https://mlsanity.com/paper/2602.01724/) (2026)
- [Bridging the Geometry Mismatch: Frequency-Aware Anisotropic Serialization for Thin-Structure SSMs](https://mlsanity.com/paper/2603.28503/) (2026)
- [Attention Is not Everything: Efficient Alternatives for Vision](https://mlsanity.com/paper/2604.17439/) (2026)
- [Advancing Vision Transformer with Enhanced Spatial Priors](https://mlsanity.com/paper/2604.18549/) (2026)
- [Increasing the Efficiency of DETR for Maritime High-Resolution Images](https://mlsanity.com/paper/2605.10269/) (2026)
- [Can Graphs Help Vision SSMs See Better?](https://mlsanity.com/paper/2605.11300/) (2026)
- [TCP-SSM: Efficient Vision State Space Models with Token-Conditioned Poles](https://mlsanity.com/paper/2605.11563/) (2026)
- [Representative Attention For Vision Transformers](https://mlsanity.com/paper/2605.14913/) (2026)
- [Cross-Modality Feature Fusion Based on Structured State Space Duality for Multimodal Image Registration Network](https://mlsanity.com/paper/2606.03341/) (2026)
- [CRISP: Calibration-Aware Visual State Space Duality for Remote Sensing Semantic Segmentation](https://mlsanity.com/paper/2608.23746/) (2026)
- [Channel-Wise and Token-Aware Post-Training Quantization for Visual State Space Duality](https://mlsanity.com/paper/2609.16656/) (2026)
- [LiAuto-MindViT: A Hybrid Vision Backbone with Adaptive Bidirectional Mamba](https://mlsanity.com/paper/2609.24337/) (2026)
- [VisionHOPE: Visual Backbones as Self-Modifying Learning Systems](https://mlsanity.com/paper/2609.33325/) (2026)
- [3D Point Tracking with State Space Models](https://mlsanity.com/paper/2609.34035/) (2026)

## Abstract (arXiv)

Vision transformers have significantly advanced the field of computer vision, offering robust modeling capabilities and global receptive field. However, their high computational demands limit their applicability in processing long sequences. To tackle this issue, State Space Models (SSMs) have gained prominence in vision tasks as they offer linear computational complexity. Recently, State Space Duality (SSD), an improved variant of SSMs, was introduced in Mamba2 to enhance model performance and efficiency. However, the inherent causal nature of SSD/SSMs restricts their applications in non-causal vision tasks. To address this limitation, we introduce Visual State Space Duality (VSSD) model, which has a non-causal format of SSD. Specifically, we propose to discard the magnitude of interactions between the hidden state and tokens while preserving their relative weights, which relieves the dependencies of token contribution on previous tokens. Together with the involvement of multi-scan strategies, we show that the scanning results can be integrated to achieve non-causality, which not only improves the performance of SSD in vision tasks but also enhances its efficiency. We conduct extensive experiments on various benchmarks including image classification, detection, and segmentation, where VSSD surpasses existing state-of-the-art SSM-based models. Code and weights are available at urlhttps://github.com/YuHengsss/VSSD.