# Doubly Deformable Aggregation of Covariance Matrices for Few-shot Segmentation

arXiv: https://arxiv.org/abs/2208.00306 · HTML: https://mlsanity.com/paper/2208.00306/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 48 references that are in the map.
- First posted on arXiv 30 Jul 2022.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Pay Attention to MLPs](https://mlsanity.com/paper/2105.08050/) (2021)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [Vision Transformer with Deformable Attention](https://mlsanity.com/paper/2201.00520/) (2022)
- [Prior Guided Feature Enrichment Network for Few-Shot Segmentation](https://mlsanity.com/paper/2008.01449/) (2020)
- [Hypercorrelation Squeeze for Few-Shot Segmentation](https://mlsanity.com/paper/2104.01538/) (2021)
- [Prototype Mixture Models for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2008.03898/) (2020)
- [Feature Weighting and Boosting for Few-Shot Segmentation](https://mlsanity.com/paper/1909.13140/) (2019)
- [SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation](https://mlsanity.com/paper/1810.09091/) (2018)
- [Part-aware Prototype Network for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2007.06309/) (2020)
- [FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation](https://mlsanity.com/paper/1907.12347/) (2019)
- [Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1407.1808/) (2014)
- [Few-Shot Segmentation Without Meta-Learning: A Good Transductive Inference Is All You Need?](https://mlsanity.com/paper/2012.06166/) (2020)
- [Few-Shot Segmentation via Cycle-Consistent Transformer](https://mlsanity.com/paper/2106.02320/) (2021)

## Built on by (most cited first, 23 of 23)

- [SegGPT: Segmenting Everything In Context](https://mlsanity.com/paper/2304.03284/) (2023)
- [RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data](https://mlsanity.com/paper/2210.12634/) (2022)
- [Neural Plasticity-Inspired Multimodal Foundation Model for Earth Observation](https://mlsanity.com/paper/2403.15356/) (2024)
- [SLiMe: Segment Like Me](https://mlsanity.com/paper/2309.03179/) (2023)
- [Self-Calibrated Cross Attention Network for Few-Shot Segmentation](https://mlsanity.com/paper/2308.09294/) (2023)
- [Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation](https://mlsanity.com/paper/2307.03407/) (2023)
- [Hybrid Mamba for Few-Shot Segmentation](https://mlsanity.com/paper/2409.19613/) (2024)
- [Eliminating Feature Ambiguity for Few-Shot Segmentation](https://mlsanity.com/paper/2407.09842/) (2024)
- [Focus on Query: Adversarial Mining Transformer for Few-Shot Segmentation](https://mlsanity.com/paper/2311.17626/) (2023)
- [Not All Data Are Unlearned Equally](https://mlsanity.com/paper/2504.05058/) (2025)
- [Relevant Intrinsic Feature Enhancement Network for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2312.06474/) (2023)
- [Unlocking the Power of SAM 2 for Few-Shot Segmentation](https://mlsanity.com/paper/2505.14100/) (2025)
- [Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation](https://mlsanity.com/paper/2504.09480/) (2025)
- [[CLS] Token is All You Need for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2304.06212/) (2023)
- [Few-shot Object Detection in Remote Sensing: Lifting the Curse of Incompletely Annotated Novel Objects](https://mlsanity.com/paper/2309.10588/) (2023)
- [CONDA: Condensed Deep Association Learning for Co-Salient Object Detection](https://mlsanity.com/paper/2409.01021/) (2024)
- [Group-On: Boosting One-Shot Segmentation with Supportive Query](https://mlsanity.com/paper/2404.11871/) (2024)
- [Beyond Mask: Rethinking Guidance Types in Few-shot Segmentation](https://mlsanity.com/paper/2407.11503/) (2024)
- [High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study](https://mlsanity.com/paper/2409.06305/) (2024)
- [Balancing Conservatism and Aggressiveness: Prototype-Affinity Hybrid Network for Few-Shot Segmentation](https://mlsanity.com/paper/2507.19140/) (2025)
- [Object-level Correlation for Few-Shot Segmentation](https://mlsanity.com/paper/2509.07917/) (2025)
- [Reflection Invariance Learning for Few-shot Semantic Segmentation](https://mlsanity.com/paper/2309.15850/) (2023)
- [GeoFlow: Geo-Aware Modeling of Inter-Area Relationships in Origin-Destination Flow Prediction and Generation](https://mlsanity.com/paper/2607.05257/) (2026)

## Abstract (arXiv)

Training semantic segmentation models with few annotated samples has great potential in various real-world applications. For the few-shot segmentation task, the main challenge is how to accurately measure the semantic correspondence between the support and query samples with limited training data. To address this problem, we propose to aggregate the learnable covariance matrices with a deformable 4D Transformer to effectively predict the segmentation map. Specifically, in this work, we first devise a novel hard example mining mechanism to learn covariance kernels for the Gaussian process. The learned covariance kernel functions have great advantages over existing cosine similarity-based methods in correspondence measurement. Based on the learned covariance kernels, an efficient doubly deformable 4D Transformer module is designed to adaptively aggregate feature similarity maps into segmentation results. By combining these two designs, the proposed method can not only set new state-of-the-art performance on public benchmarks, but also converge extremely faster than existing methods. Experiments on three public datasets have demonstrated the effectiveness of our method.