# Few-Shot Segmentation via Cycle-Consistent Transformer

arXiv: https://arxiv.org/abs/2106.02320 · HTML: https://mlsanity.com/paper/2106.02320/

## Key facts

- Cited by 133 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 52 references that are in the map.
- First posted on arXiv 4 Jun 2021.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [Prior Guided Feature Enrichment Network for Few-Shot Segmentation](https://mlsanity.com/paper/2008.01449/) (2020)
- [Learning Correspondence from the Cycle-Consistency of Time](https://mlsanity.com/paper/1903.07593/) (2019)
- [CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning](https://mlsanity.com/paper/1903.02351/) (2019)
- [Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach](https://mlsanity.com/paper/1703.08448/) (2017)
- [Hypercorrelation Squeeze for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01538/) (2021)
- [Prototype Mixture Models for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2008.03898/) (2020)

## Built on by (most cited first, 30 of 133)

- [Visual Prompting via Image Inpainting](https://mlsanity.com/paper/2209.00647/) (2022)
- [SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model](https://mlsanity.com/paper/2303.05118/) (2023)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2207.10866/) (2022)
- [Hierarchical Dense Correlation Distillation for Few-Shot Segmentation](https://mlsanity.com/paper/2303.14652/) (2023)
- [VRP-SAM: SAM with Visual Reference Prompt](https://mlsanity.com/paper/2402.17726/) (2024)
- [Dense Cross-Query-and-Support Attention Weighted Mask Aggregation for Few-Shot Segmentation](https://mlsanity.com/paper/2207.08549/) (2022)
- [Feature-Proxy Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2210.06908/) (2022)
- [Learning Mask-aware CLIP Representations for Zero-Shot Segmentation](https://mlsanity.com/paper/2310.00240/) (2023)
- [Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning](https://mlsanity.com/paper/2206.06122/) (2022)
- [Exploring Effective Factors for Improving Visual In-Context Learning](https://mlsanity.com/paper/2304.04748/) (2023)
- [Intermediate Prototype Mining Transformer for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2210.06780/) (2022)
- [Bridge the Points: Graph-based Few-shot Segment Anything Semantically](https://mlsanity.com/paper/2410.06964/) (2024)
- [Few-shot Medical Image Segmentation with Cycle-resemblance Attention](https://mlsanity.com/paper/2212.03967/) (2022)
- [Self-Calibrated Cross Attention Network for Few-Shot Segmentation](https://mlsanity.com/paper/2308.09294/) (2023)
- [One-Shot Transfer of Affordance Regions? AffCorrs!](https://mlsanity.com/paper/2209.07147/) (2022)
- [Cross-Domain Few-Shot Segmentation via Iterative Support-Query Correspondence Mining](https://mlsanity.com/paper/2401.08407/) (2024)
- [APSeg: Auto-Prompt Network for Cross-Domain Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2406.08372/) (2024)
- [MIANet: Aggregating Unbiased Instance and General Information for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2305.13864/) (2023)
- [Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation](https://mlsanity.com/paper/2307.03407/) (2023)
- [Doubly Deformable Aggregation of Covariance Matrices for Few-shot Segmentation](https://mlsanity.com/paper/2208.00306/) (2022)
- [A Strong Baseline for Generalized Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2211.14126/) (2022)
- [Mask Matching Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2301.01208/) (2022)
- [DifFSS: Diffusion Model for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2307.00773/) (2023)
- [Multi-Modal Prototypes for Open-World Semantic Segmentation](https://mlsanity.com/paper/2307.02003/) (2023)
- [Adapt Before Comparison: A New Perspective on Cross-Domain Few-Shot Segmentation](https://mlsanity.com/paper/2402.17614/) (2024)
- [MSANet: Multi-Similarity and Attention Guidance for Boosting Few-Shot Segmentation](https://mlsanity.com/paper/2206.09667/) (2022)
- [Hybrid Mamba for Few-Shot Segmentation](https://mlsanity.com/paper/2409.19613/) (2024)
- [Few Shot Medical Image Segmentation with Cross Attention Transformer](https://mlsanity.com/paper/2303.13867/) (2023)

## Abstract (arXiv)

Few-shot segmentation aims to train a segmentation model that can fast adapt to novel classes with few exemplars. The conventional training paradigm is to learn to make predictions on query images conditioned on the features from support images. Previous methods only utilized the semantic-level prototypes of support images as conditional information. These methods cannot utilize all pixel-wise support information for the query predictions, which is however critical for the segmentation task. In this paper, we focus on utilizing pixel-wise relationships between support and query images to facilitate the few-shot segmentation task. We design a novel Cycle-Consistent TRansformer (CyCTR) module to aggregate pixel-wise support features into query ones. CyCTR performs cross-attention between features from different images, i.e. support and query images. We observe that there may exist unexpected irrelevant pixel-level support features. Directly performing cross-attention may aggregate these features from support to query and bias the query features. Thus, we propose using a novel cycle-consistent attention mechanism to filter out possible harmful support features and encourage query features to attend to the most informative pixels from support images. Experiments on all few-shot segmentation benchmarks demonstrate that our proposed CyCTR leads to remarkable improvement compared to previous state-of-the-art methods. Specifically, on Pascal-5i and COCO-20i datasets, we achieve 67.5% and 45.6% mIoU for 5-shot segmentation, outperforming previous state-of-the-art methods by 5.6% and 7.1% respectively.