# Vision Graph Prompting via Semantic Low-Rank Decomposition

arXiv: https://arxiv.org/abs/2505.04121 · HTML: https://mlsanity.com/paper/2505.04121/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 7 May 2025.

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [MoleculeNet: A Benchmark for Molecular Machine Learning](https://mlsanity.com/paper/1703.00564/) (2017)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [DeepGCNs: Can GCNs Go as Deep as CNNs?](https://mlsanity.com/paper/1904.03751/) (2019)
- [Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud](https://mlsanity.com/paper/2003.01251/) (2020)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [Vision GNN: An Image is Worth Graph of Nodes](https://mlsanity.com/paper/2206.00272/) (2022)
- [Universal Prompt Tuning for Graph Neural Networks](https://mlsanity.com/paper/2209.15240/) (2022)
- [E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning](https://mlsanity.com/paper/2307.13770/) (2023)
- [Diversity-Aware Meta Visual Prompting](https://mlsanity.com/paper/2303.08138/) (2023)
- [MobileViG: Graph-Based Sparse Attention for Mobile Vision Applications](https://mlsanity.com/paper/2307.00395/) (2023)
- [Visual Fourier Prompt Tuning](https://mlsanity.com/paper/2411.01327/) (2024)
- [GreedyViG: Dynamic Axial Graph Construction for Efficient Vision GNNs](https://mlsanity.com/paper/2405.06849/) (2024)
- [STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding](https://mlsanity.com/paper/2503.15973/) (2025)
- [CAPrompt: Cyclic Prompt Aggregation for Pre-Trained Model Based Class Incremental Learning](https://mlsanity.com/paper/2412.08929/) (2024)
- [Selective Visual Prompting in Vision Mamba](https://mlsanity.com/paper/2412.08947/) (2024)

## Built on by (most cited first, 3 of 3)

- [UPP: Unified Point-Level Prompting for Robust Point Cloud Analysis](https://mlsanity.com/paper/2507.18997/) (2025)
- [Unified Graph Prompt Learning via Low-Rank Graph Message Prompting](https://mlsanity.com/paper/2604.11257/) (2026)
- [Task-Specific Prompt with Global Context for Multi-Task Graph Pre-Training](https://mlsanity.com/paper/2609.00047/) (2026)

## Abstract (arXiv)

Vision GNN (ViG) demonstrates superior performance by representing images as graph structures, providing a more natural way to capture irregular semantic patterns beyond traditional grid or sequence-based representations. To efficiently adapt ViG to downstream tasks, parameter-efficient fine-tuning techniques like visual prompting become increasingly essential. However, existing prompting methods are primarily designed for Transformer-based models, neglecting the rich topological relationships among nodes and edges in graph-based representations, limiting their capacity to model complex semantics. In this paper, we propose Vision Graph Prompting (VGP), a novel framework tailored for vision graph structures. Our core insight reveals that semantically connected components in the graph exhibit low-rank properties. Building on this observation, we introduce a semantic low-rank prompting method that decomposes low-rank semantic features and integrates them with prompts on vision graph topologies, capturing both global structural patterns and fine-grained semantic dependencies. Extensive experiments demonstrate our method significantly improves ViG's transfer performance on diverse downstream tasks, achieving results comparable to full fine-tuning while maintaining parameter efficiency. Our code is available at https://github.com/zhoujiahuan1991/ICML2025-VGP.