# Selective Visual Prompting in Vision Mamba

arXiv: https://arxiv.org/abs/2412.08947 · HTML: https://mlsanity.com/paper/2412.08947/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 17 of its 36 references that are in the map.
- First posted on arXiv 12 Dec 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Vision Mamba backbones](https://mlsanity.com/ssm/vision-mamba-backbones/)

## Builds on (most cited first, 17 of 17)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning](https://mlsanity.com/paper/2206.06522/) (2022)
- [Three things everyone should know about Vision Transformers](https://mlsanity.com/paper/2203.09795/) (2022)
- [VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding](https://mlsanity.com/paper/2312.08733/) (2023)
- [Efficient Adaptation of Large Vision Transformer via Adapter Re-Composing](https://mlsanity.com/paper/2310.06234/) (2023)
- [Revisiting the Power of Prompt for Visual Tuning](https://mlsanity.com/paper/2402.02382/) (2024)
- [AutoVP: An Automated Visual Prompting Framework and Benchmark](https://mlsanity.com/paper/2310.08381/) (2023)

## Built on by (most cited first, 6 of 6)

- [Token Coordinated Prompt Attention is Needed for Visual Prompting](https://mlsanity.com/paper/2505.02406/) (2025)
- [Vision Graph Prompting via Semantic Low-Rank Decomposition](https://mlsanity.com/paper/2505.04121/) (2025)
- [State Space Prompting via Gathering and Spreading Spatio-Temporal Information for Video Understanding](https://mlsanity.com/paper/2510.12160/) (2025)
- [Partial Ring Scan: Revisiting Scan Order in Vision State Space Models](https://mlsanity.com/paper/2602.04170/) (2026)
- [Beyond BEV: Optimizing Point-Level Tokens for Collaborative Perception](https://mlsanity.com/paper/2508.19638/) (2025)
- [CKDA: Cross-modality Knowledge Disentanglement and Alignment for Visible-Infrared Lifelong Person Re-identification](https://mlsanity.com/paper/2511.15016/) (2025)

## Abstract (arXiv)

Pre-trained Vision Mamba (Vim) models have demonstrated exceptional performance across various computer vision tasks in a computationally efficient manner, attributed to their unique design of selective state space models. To further extend their applicability to diverse downstream vision tasks, Vim models can be adapted using the efficient fine-tuning technique known as visual prompting. However, existing visual prompting methods are predominantly tailored for Vision Transformer (ViT)-based models that leverage global attention, neglecting the distinctive sequential token-wise compression and propagation characteristics of Vim. Specifically, existing prompt tokens prefixed to the sequence are insufficient to effectively activate the input and forget gates across the entire sequence, hindering the extraction and propagation of discriminative information. To address this limitation, we introduce a novel Selective Visual Prompting (SVP) method specifically for the efficient fine-tuning of Vim. To prevent the loss of discriminative information during state space propagation, SVP employs lightweight selective prompters for token-wise prompt generation, ensuring adaptive activation of the update and forget gates within Mamba blocks to promote discriminative information propagation. Moreover, considering that Vim propagates both shared cross-layer information and specific inner-layer information, we further refine SVP with a dual-path structure: Cross-Prompting and Inner-Prompting. Cross-Prompting utilizes shared parameters across layers, while Inner-Prompting employs distinct parameters, promoting the propagation of both shared and specific information, respectively. Extensive experimental results on various large-scale benchmarks demonstrate that our proposed SVP significantly outperforms state-of-the-art methods. Our code is available at https://github.com/zhoujiahuan1991/AAAI2025-SVP.