# Label Anything: Multi-Class Few-Shot Semantic Segmentation with Visual Prompts

arXiv: https://arxiv.org/abs/2407.02075 · HTML: https://mlsanity.com/paper/2407.02075/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 45 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 30 of 38)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Prior Guided Feature Enrichment Network for Few-Shot Segmentation](https://mlsanity.com/paper/2008.01449/) (2020)
- [Inpaint Anything: Segment Anything Meets Image Inpainting](https://mlsanity.com/paper/2304.06790/) (2023)
- [CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning](https://mlsanity.com/paper/1903.02351/) (2019)
- [Adaptive Prototype Learning and Allocation for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01893/) (2021)
- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [Hypercorrelation Squeeze for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01538/) (2021)
- [Feature Weighting and Boosting for Few-Shot Segmentation](https://mlsanity.com/paper/1909.13140/) (2019)
- [SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation](https://mlsanity.com/paper/1810.09091/) (2018)
- [Part-aware Prototype Network for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2007.06309/) (2020)
- [Few-Shot Segmentation via Cycle-Consistent Transformer](https://mlsanity.com/paper/2106.02320/) (2021)

## Built on by (most cited first, 9 of 9)

- [MultiverSeg: Scalable Interactive Segmentation of Biomedical Imaging Datasets with In-Context Guidance](https://mlsanity.com/paper/2412.15058/) (2024)
- [Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challenges](https://mlsanity.com/paper/2507.09562/) (2025)
- [Revealing the Semantic Selection Gap in DINOv3 through Training-Free Few-Shot Segmentation](https://mlsanity.com/paper/2602.07550/) (2026)
- [Take a Peek: Efficient Encoder Adaptation for Few-Shot Semantic Segmentation via LoRA](https://mlsanity.com/paper/2512.10521/) (2025)
- [Segment Any Class (SAC): Multi-Class Few-Shot Semantic Segmentation via Class Region Proposals](https://mlsanity.com/paper/2411.13774/) (2024)
- [Matching-Based Few-Shot Semantic Segmentation Models Are Interpretable by Design](https://mlsanity.com/paper/2511.18163/) (2025)
- [Boundary-Aware Test-Time Adaptation for Zero-Shot Medical Image Segmentation](https://mlsanity.com/paper/2512.04520/) (2025)
- [DistillFSS: Synthesizing Few-Shot Knowledge into a Lightweight Segmentation Model](https://mlsanity.com/paper/2512.05613/) (2025)
- [SADe: Sparse-Atom Support Decontamination for Few-Shot Segmentation with Weak Support Annotations](https://mlsanity.com/paper/2607.24706/) (2026)

## Abstract (arXiv)

Few-shot semantic segmentation aims to segment objects from previously unseen classes using only a limited number of labeled examples. In this paper, we introduce Label Anything, a novel transformer-based architecture designed for multi-prompt, multi-way few-shot semantic segmentation. Our approach leverages diverse visual prompts -- points, bounding boxes, and masks -- to create a highly flexible and generalizable framework that significantly reduces annotation burden while maintaining high accuracy. Label Anything makes three key contributions: (textiti) we introduce a new task formulation that relaxes conventional few-shot segmentation constraints by supporting various types of prompts, multi-class classification, and enabling multiple prompts within a single image; (textitii) we propose a novel architecture based on transformers and attention mechanisms; and (textitiii) we design a versatile training procedure allowing our model to operate seamlessly across different N-way K-shot and prompt-type configurations with a single trained model. Our extensive experimental evaluation on the widely used COCO-20i benchmark demonstrates that Label Anything achieves state-of-the-art performance among existing multi-way few-shot segmentation methods, while significantly outperforming leading single-class models when evaluated in multi-class settings. Code and trained models are available at https://github.com/pasqualedem/LabelAnything.