# Cost Aggregation Is All You Need for Few-Shot Segmentation

arXiv: https://arxiv.org/abs/2112.11685 · HTML: https://mlsanity.com/paper/2112.11685/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 87 references that are in the map.
- First posted on arXiv 22 Dec 2021.

## Builds on (most cited first, 30 of 64)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Albumentations: fast and flexible image augmentations](https://mlsanity.com/paper/1809.06839/) (2018)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [Early Convolutions Help Transformers See Better](https://mlsanity.com/paper/2106.14881/) (2021)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [Relation Networks for Object Detection](https://mlsanity.com/paper/1711.11575/) (2017)
- [Prior Guided Feature Enrichment Network for Few-Shot Segmentation](https://mlsanity.com/paper/2008.01449/) (2020)
- [CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning](https://mlsanity.com/paper/1903.02351/) (2019)
- [Local Relation Networks for Image Recognition](https://mlsanity.com/paper/1904.11491/) (2019)
- [Adaptive Prototype Learning and Allocation for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01893/) (2021)
- [Hypercorrelation Squeeze for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01538/) (2021)
- [Prototype Mixture Models for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2008.03898/) (2020)
- [Feature Weighting and Boosting for Few-Shot Segmentation](https://mlsanity.com/paper/1909.13140/) (2019)

## Built on by (most cited first, 16 of 16)

- [Feature-Proxy Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2210.06908/) (2022)
- [Learning Semantic Correspondence with Sparse Annotations](https://mlsanity.com/paper/2208.06974/) (2022)
- [Doubly Deformable Aggregation of Covariance Matrices for Few-shot Segmentation](https://mlsanity.com/paper/2208.00306/) (2022)
- [Multi-Modal Prototypes for Open-World Semantic Segmentation](https://mlsanity.com/paper/2307.02003/) (2023)
- [MSANet: Multi-Similarity and Attention Guidance for Boosting Few-Shot Segmentation](https://mlsanity.com/paper/2206.09667/) (2022)
- [MSI: Maximize Support-Set Information for Few-Shot Segmentation](https://mlsanity.com/paper/2212.04673/) (2022)
- [HM: Hybrid Masking for Few-Shot Segmentation](https://mlsanity.com/paper/2203.12826/) (2022)
- [SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation](https://mlsanity.com/paper/2505.21795/) (2025)
- [Integrative Feature and Cost Aggregation with Transformers for Dense Correspondence](https://mlsanity.com/paper/2209.08742/) (2022)
- [SAMIC: Segment Anything with In-Context Spatial Prompt Engineering](https://mlsanity.com/paper/2412.11998/) (2024)
- [Elimination of Non-Novel Segments at Multi-Scale for Few-Shot Segmentation](https://mlsanity.com/paper/2211.02300/) (2022)
- [Multi-similarity based Hyperrelation Network for few-shot segmentation](https://mlsanity.com/paper/2203.09550/) (2022)
- [Background Clustering Pre-training for Few-shot Segmentation](https://mlsanity.com/paper/2312.03322/) (2023)
- [FCC: Fully Connected Correlation for One-Shot Segmentation](https://mlsanity.com/paper/2411.11917/) (2024)
- [Static Segmentation by Tracking: A Label-Efficient Approach for Fine-Grained Specimen Image Segmentation](https://mlsanity.com/paper/2501.06749/) (2025)
- [Inter- and Intra-image Refinement for Few Shot Segmentation](https://mlsanity.com/paper/2507.05838/) (2025)

## Abstract (arXiv)

We introduce a novel cost aggregation network, dubbed Volumetric Aggregation with Transformers (VAT), to tackle the few-shot segmentation task by using both convolutions and transformers to efficiently handle high dimensional correlation maps between query and support. In specific, we propose our encoder consisting of volume embedding module to not only transform the correlation maps into more tractable size but also inject some convolutional inductive bias and volumetric transformer module for the cost aggregation. Our encoder has a pyramidal structure to let the coarser level aggregation to guide the finer level and enforce to learn complementary matching scores. We then feed the output into our affinity-aware decoder along with the projected feature maps for guiding the segmentation process. Combining these components, we conduct experiments to demonstrate the effectiveness of the proposed method, and our method sets a new state-of-the-art for all the standard benchmarks in few-shot segmentation task. Furthermore, we find that the proposed method attains state-of-the-art performance even for the standard benchmarks in semantic correspondence task although not specifically designed for this task. We also provide an extensive ablation study to validate our architectural choices. The trained weights and codes are available at: https://seokju-cho.github.io/VAT/.