# Feature-Proxy Transformer for Few-Shot Segmentation

arXiv: https://arxiv.org/abs/2210.06908 · HTML: https://mlsanity.com/paper/2210.06908/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 76 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [Prior Guided Feature Enrichment Network for Few-Shot Segmentation](https://mlsanity.com/paper/2008.01449/) (2020)
- [One-Shot Video Object Segmentation](https://mlsanity.com/paper/1611.05198/) (2016)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://mlsanity.com/paper/2109.04332/) (2021)
- [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018)
- [Associating Objects with Transformers for Video Object Segmentation](https://mlsanity.com/paper/2106.02638/) (2021)
- [CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning](https://mlsanity.com/paper/1903.02351/) (2019)
- [Adaptive Prototype Learning and Allocation for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01893/) (2021)
- [Hypercorrelation Squeeze for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01538/) (2021)

## Built on by (most cited first, 30 of 58)

- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching](https://mlsanity.com/paper/2305.13310/) (2023)
- [VRP-SAM: SAM with Visual Reference Prompt](https://mlsanity.com/paper/2402.17726/) (2024)
- [Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception](https://mlsanity.com/paper/2403.02969/) (2024)
- [Bridge the Points: Graph-based Few-shot Segment Anything Semantically](https://mlsanity.com/paper/2410.06964/) (2024)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [SLiMe: Segment Like Me](https://mlsanity.com/paper/2309.03179/) (2023)
- [Few-Shot Medical Image Segmentation via a Region-enhanced Prototypical Transformer](https://mlsanity.com/paper/2309.04825/) (2023)
- [Multi-Modal Prototypes for Open-World Semantic Segmentation](https://mlsanity.com/paper/2307.02003/) (2023)
- [SEGIC: Unleashing the Emergent Correspondence for In-Context Segmentation](https://mlsanity.com/paper/2311.14671/) (2023)
- [A Simple Image Segmentation Framework via In-Context Examples](https://mlsanity.com/paper/2410.04842/) (2024)
- [Lightweight Frequency Masker for Cross-Domain Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2410.22135/) (2024)
- [Prediction Calibration for Generalized Few-shot Semantic Segmentation](https://mlsanity.com/paper/2210.08290/) (2022)
- [Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation](https://mlsanity.com/paper/2410.02369/) (2024)
- [Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach](https://mlsanity.com/paper/2404.11732/) (2024)
- [DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistency](https://mlsanity.com/paper/2504.12080/) (2025)
- [Domain-Rectifying Adapter for Cross-Domain Few-Shot Segmentation](https://mlsanity.com/paper/2404.10322/) (2024)
- [Focus on Query: Adversarial Mining Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2311.17626/) (2023)
- [Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation](https://mlsanity.com/paper/2301.01156/) (2023)
- [Explore In-Context Segmentation via Latent Diffusion Models](https://mlsanity.com/paper/2403.09616/) (2024)
- [Not Just Learning from Others but Relying on Yourself: A New Perspective on Few-Shot Segmentation in Remote Sensing](https://mlsanity.com/paper/2310.12452/) (2023)
- [Towards More Unified In-context Visual Understanding](https://mlsanity.com/paper/2312.02520/) (2023)
- [Prompt-and-Transfer: Dynamic Class-aware Enhancement for Few-shot Segmentation](https://mlsanity.com/paper/2409.10389/) (2024)
- [Partition-A-Medical-Image: Extracting Multiple Representative Sub-regions for Few-shot Medical Image Segmentation](https://mlsanity.com/paper/2309.11172/) (2023)
- [Label Anything: Multi-Class Few-Shot Semantic Segmentation with Visual Prompts](https://mlsanity.com/paper/2407.02075/) (2024)
- [The Devil is in Low-Level Features for Cross-Domain Few-Shot Segmentation](https://mlsanity.com/paper/2503.21150/) (2025)
- [Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2506.07376/) (2025)
- [AgMTR: Agent Mining Transformer for Few-shot Segmentation in Remote Sensing](https://mlsanity.com/paper/2409.17453/) (2024)
- [RobustEMD: Domain Robust Matching for Cross-domain Few-shot Medical Image Segmentation](https://mlsanity.com/paper/2410.01110/) (2024)
- [[CLS] Token is All You Need for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2304.06212/) (2023)

## Abstract (arXiv)

Few-shot segmentation (FSS) aims at performing semantic segmentation on novel classes given a few annotated support samples. With a rethink of recent advances, we find that the current FSS framework has deviated far from the supervised segmentation framework: Given the deep features, FSS methods typically use an intricate decoder to perform sophisticated pixel-wise matching, while the supervised segmentation methods use a simple linear classification head. Due to the intricacy of the decoder and its matching pipeline, it is not easy to follow such an FSS framework. This paper revives the straightforward framework of "feature extractor + linear classification head" and proposes a novel Feature-Proxy Transformer (FPTrans) method, in which the "proxy" is the vector representing a semantic class in the linear classification head. FPTrans has two keypoints for learning discriminative features and representative proxies: 1) To better utilize the limited support samples, the feature extractor makes the query interact with the support features from the bottom to top layers using a novel prompting strategy. 2) FPTrans uses multiple local background proxies (instead of a single one) because the background is not homogeneous and may contain some novel foreground regions. These two keypoints are easily integrated into the vision transformer backbone with the prompting mechanism in the transformer. Given the learned features and proxies, FPTrans directly compares their cosine similarity for segmentation. Although the framework is straightforward, we show that FPTrans achieves competitive FSS accuracy on par with state-of-the-art decoder-based methods.