# Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer

arXiv: https://arxiv.org/abs/2108.03032 · HTML: https://mlsanity.com/paper/2108.03032/

## Key facts

- Cited by 116 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 42 references that are in the map.
- First posted on arXiv 6 Aug 2021.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions](https://mlsanity.com/paper/1812.03664/) (2018)
- [Prior Guided Feature Enrichment Network for Few-Shot Segmentation](https://mlsanity.com/paper/2008.01449/) (2020)
- [CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning](https://mlsanity.com/paper/1903.02351/) (2019)
- [Prototype Mixture Models for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2008.03898/) (2020)
- [Feature Weighting and Boosting for Few-Shot Segmentation](https://mlsanity.com/paper/1909.13140/) (2019)
- [SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation](https://mlsanity.com/paper/1810.09091/) (2018)
- [Part-aware Prototype Network for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2007.06309/) (2020)
- [Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1407.1808/) (2014)

## Built on by (most cited first, 30 of 116)

- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Task Residual for Tuning Vision-Language Models](https://mlsanity.com/paper/2211.10277/) (2022)
- [Learning What Not to Segment: A New Perspective on Few-Shot Segmentation](https://mlsanity.com/paper/2203.07615/) (2022)
- [Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2207.10866/) (2022)
- [Self-Support Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2207.11549/) (2022)
- [Hierarchical Dense Correlation Distillation for Few-Shot Segmentation](https://mlsanity.com/paper/2303.14652/) (2023)
- [VRP-SAM: SAM with Visual Reference Prompt](https://mlsanity.com/paper/2402.17726/) (2024)
- [GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph](https://mlsanity.com/paper/2309.13625/) (2023)
- [Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2203.04845/) (2022)
- [Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging](https://mlsanity.com/paper/2205.10102/) (2022)
- [Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentation](https://mlsanity.com/paper/2207.08549/) (2022)
- [Feature-Proxy Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2210.06908/) (2022)
- [Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning](https://mlsanity.com/paper/2206.06122/) (2022)
- [Open-vocabulary Semantic Segmentation with Frozen Vision-Language Models](https://mlsanity.com/paper/2210.15138/) (2022)
- [Intermediate Prototype Mining Transformer for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2210.06780/) (2022)
- [Compound Prototype Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2207.05515/) (2022)
- [Self-Distilled Vision Transformer for Domain Generalization](https://mlsanity.com/paper/2207.12392/) (2022)
- [PFENet++: Boosting Few-shot Semantic Segmentation with the Noise-filtered Context-aware Prior Mask](https://mlsanity.com/paper/2109.13788/) (2021)
- [Cross-Domain Few-Shot Segmentation via Iterative Support-Query Correspondence Mining](https://mlsanity.com/paper/2401.08407/) (2024)
- [Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation](https://mlsanity.com/paper/2204.09903/) (2022)
- [Prototype Adaption and Projection for Few- and Zero-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2305.14335/) (2023)
- [APSeg: Auto-Prompt Network for Cross-Domain Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2406.08372/) (2024)
- [Interclass Prototype Relation for Few-Shot Segmentation](https://mlsanity.com/paper/2211.08681/) (2022)
- [MIANet: Aggregating Unbiased Instance and General Information for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2305.13864/) (2023)
- [A Strong Baseline for Generalized Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2211.14126/) (2022)
- [Mask Matching Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2301.01208/) (2022)
- [FECANet: Boosting Few-Shot Semantic Segmentation with Feature-Enhanced Context-Aware Network](https://mlsanity.com/paper/2301.08160/) (2023)
- [Multi-Modal Prototypes for Open-World Semantic Segmentation](https://mlsanity.com/paper/2307.02003/) (2023)
- [Adapt Before Comparison: A New Perspective on Cross-Domain Few-Shot Segmentation](https://mlsanity.com/paper/2402.17614/) (2024)

## Abstract (arXiv)

A few-shot semantic segmentation model is typically composed of a CNN encoder, a CNN decoder and a simple classifier (separating foreground and background pixels). Most existing methods meta-learn all three model components for fast adaptation to a new class. However, given that as few as a single support set image is available, effective model adaption of all three components to the new class is extremely challenging. In this work we propose to simplify the meta-learning task by focusing solely on the simplest component, the classifier, whilst leaving the encoder and decoder to pre-training. We hypothesize that if we pre-train an off-the-shelf segmentation model over a set of diverse training classes with sufficient annotations, the encoder and decoder can capture rich discriminative features applicable for any unseen classes, rendering the subsequent meta-learning stage unnecessary. For the classifier meta-learning, we introduce a Classifier Weight Transformer (CWT) designed to dynamically adapt the supportset trained classifier's weights to each query image in an inductive way. Extensive experiments on two standard benchmarks show that despite its simplicity, our method outperforms the state-of-the-art alternatives, often by a large margin.Code is available on https://github.com/zhiheLu/CWT-for-FSS.