# Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach

arXiv: https://arxiv.org/abs/2404.11732 · HTML: https://mlsanity.com/paper/2404.11732/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 52 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Meta-Learning for Semi-Supervised Few-Shot Classification](https://mlsanity.com/paper/1803.00676/) (2018)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [A Baseline for Few-Shot Image Classification](https://mlsanity.com/paper/1909.02729/) (2019)
- [Frustratingly Simple Few-Shot Object Detection](https://mlsanity.com/paper/2003.06957/) (2020)
- [Prior Guided Feature Enrichment Network for Few-Shot Segmentation](https://mlsanity.com/paper/2008.01449/) (2020)
- [Few-shot Object Detection via Feature Reweighting](https://mlsanity.com/paper/1812.01866/) (2018)
- [CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning](https://mlsanity.com/paper/1903.02351/) (2019)
- [Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning](https://mlsanity.com/paper/1805.10002/) (2018)
- [Hypercorrelation Squeeze for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01538/) (2021)
- [Modeling the Background for Incremental Learning in Semantic Segmentation](https://mlsanity.com/paper/2002.00718/) (2020)
- [Prototype Mixture Models for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2008.03898/) (2020)
- [Meta R-CNN : Towards General Solver for Instance-level Few-shot Learning](https://mlsanity.com/paper/1909.13032/) (2019)
- [Part-aware Prototype Network for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2007.06309/) (2020)

## Built on by (most cited first, 15 of 15)

- [Prompt-based Adaptation in Large-scale Vision Models: A Survey](https://mlsanity.com/paper/2510.13219/) (2025)
- [Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Model](https://mlsanity.com/paper/2503.16282/) (2025)
- [DPSeg: Dual-Prompt Cost Volume Learning for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2505.11676/) (2025)
- [Forget Vectors at Play: Universal Input Perturbations Driving Machine Unlearning in Image Classification](https://mlsanity.com/paper/2412.16780/) (2024)
- [MedVisionLlama: Leveraging Pre-Trained Large Language Model Layers to Enhance Medical Image Segmentation](https://mlsanity.com/paper/2410.02458/) (2024)
- [The Power of One: A Single Example is All it Takes for Segmentation in VLMs](https://mlsanity.com/paper/2503.10779/) (2025)
- [From Ideal to Real: Unified and Data-Efficient Dense Prediction for Real-World Scenarios](https://mlsanity.com/paper/2506.20279/) (2025)
- [Does GenAI Rewrite How We Write? An Empirical Study on Two-Million Preprints](https://mlsanity.com/paper/2510.17882/) (2025)
- [Omni-Referring Image Segmentation](https://mlsanity.com/paper/2512.06862/) (2025)
- [Probabilistic Prototype Calibration of Vision-Language Models for Generalized Few-shot Semantic Segmentation](https://mlsanity.com/paper/2506.22979/) (2025)
- [Make It Up: Fake Images, Real Gains in Generalized Few-shot Semantic Segmentation](https://mlsanity.com/paper/2603.27206/) (2026)
- [FlexICL: A Flexible Visual In-context Learning Framework for Elbow and Wrist Ultrasound Segmentation](https://mlsanity.com/paper/2510.26049/) (2025)
- [Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?](https://mlsanity.com/paper/2602.23339/) (2026)
- [Training-Free Generalized Few-Shot Segmentation through Open-Vocabulary Semantic Arbitration](https://mlsanity.com/paper/2606.09474/) (2026)
- [SOS! : A Streamlined Object-Conditional Transformer for Model-free Segmentation](https://mlsanity.com/paper/2608.15295/) (2026)

## Abstract (arXiv)

The emergence of attention-based transformer models has led to their extensive use in various tasks, due to their superior generalization and transfer properties. Recent research has demonstrated that such models, when prompted appropriately, are excellent for few-shot inference. However, such techniques are under-explored for dense prediction tasks like semantic segmentation. In this work, we examine the effectiveness of prompting a transformer-decoder with learned visual prompts for the generalized few-shot segmentation (GFSS) task. Our goal is to achieve strong performance not only on novel categories with limited examples, but also to retain performance on base categories. We propose an approach to learn visual prompts with limited examples. These learned visual prompts are used to prompt a multiscale transformer decoder to facilitate accurate dense predictions. Additionally, we introduce a unidirectional causal attention mechanism between the novel prompts, learned with limited examples, and the base prompts, learned with abundant data. This mechanism enriches the novel prompts without deteriorating the base class performance. Overall, this form of prompting helps us achieve state-of-the-art performance for GFSS on two different benchmark datasets: COCO-20i and Pascal-5i, without the need for test-time optimization (or transduction). Furthermore, test-time optimization leveraging unlabelled test data can be used to improve the prompts, which we refer to as transductive prompt tuning.