# CrossTransformers: spatially-aware few-shot transfer

arXiv: https://arxiv.org/abs/2007.11498 · HTML: https://mlsanity.com/paper/2007.11498/

## Key facts

- Cited by 174 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2020.

## Built on by (most cited first, 30 of 174)

- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers](https://mlsanity.com/paper/2106.05392/) (2021)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference](https://mlsanity.com/paper/2204.07305/) (2022)
- [Are Large-scale Datasets Necessary for Self-Supervised Pre-training?](https://mlsanity.com/paper/2112.10740/) (2021)
- [Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation](https://mlsanity.com/paper/2107.03358/) (2021)
- [R2Former: Unified Retrieval and Reranking Transformer for Place Recognition](https://mlsanity.com/paper/2304.03410/) (2023)
- [Self-Support Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2207.11549/) (2022)
- [Representational Continuity for Unsupervised Continual Learning](https://mlsanity.com/paper/2110.06976/) (2021)
- [Watermarking Images in Self-Supervised Latent Spaces](https://mlsanity.com/paper/2112.09581/) (2021)
- [Temporal-Relational CrossTransformers for Few-Shot Action Recognition](https://mlsanity.com/paper/2101.06184/) (2021)
- [Cross-domain Few-shot Learning with Task-specific Adapters](https://mlsanity.com/paper/2107.00358/) (2021)
- [Relational Embedding for Few-Shot Classification](https://mlsanity.com/paper/2108.09666/) (2021)
- [Few-Shot Object Detection with Fully Cross-Transformer](https://mlsanity.com/paper/2203.15021/) (2022)
- [Where in the World is this Image? Transformer-based Geo-localization in the Wild](https://mlsanity.com/paper/2204.13861/) (2022)
- [Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning](https://mlsanity.com/paper/2103.01315/) (2021)
- [Universal Representation Learning from Multiple Domains for Few-shot Classification](https://mlsanity.com/paper/2103.13841/) (2021)
- [Spatio-temporal Relation Modeling for Few-shot Action Recognition](https://mlsanity.com/paper/2112.05132/) (2021)
- [ISTR: End-to-End Instance Segmentation with Transformers](https://mlsanity.com/paper/2105.00637/) (2021)
- [Rectifying the Shortcut Learning of Background for Few-Shot Learning](https://mlsanity.com/paper/2107.07746/) (2021)
- [Joint Distribution Matters: Deep Brownian Distance Covariance for Few-Shot Classification](https://mlsanity.com/paper/2204.04567/) (2022)
- [A Closer Look at Few-shot Classification Again](https://mlsanity.com/paper/2301.12246/) (2023)
- [Label, Verify, Correct: A Simple Few Shot Object Detection Method](https://mlsanity.com/paper/2112.05749/) (2021)
- [Rethinking Generalization in Few-Shot Classification](https://mlsanity.com/paper/2206.07267/) (2022)
- [Compound Prototype Matching for Few-shot Action Recognition](https://mlsanity.com/paper/2207.05515/) (2022)
- [AlignMixup: Improving Representations By Interpolating Aligned Features](https://mlsanity.com/paper/2103.15375/) (2021)
- [INTERN: A New Learning Paradigm Towards General Vision](https://mlsanity.com/paper/2111.08687/) (2021)

## Abstract (arXiv)

Given new tasks with very little data-such as new classes in a classification problem or a domain shift in the input-performance of modern vision systems degrades remarkably quickly. In this work, we illustrate how the neural network representations which underpin modern vision systems are subject to supervision collapse, whereby they lose any information that is not necessary for performing the training task, including information that may be necessary for transfer to new tasks or domains. We then propose two methods to mitigate this problem. First, we employ self-supervised learning to encourage general-purpose features that transfer better. Second, we propose a novel Transformer based neural network architecture called CrossTransformers, which can take a small number of labeled images and an unlabeled query, find coarse spatial correspondence between the query and the labeled images, and then infer class membership by computing distances between spatially-corresponding features. The result is a classifier that is more robust to task and domain shift, which we demonstrate via state-of-the-art performance on Meta-Dataset, a recent dataset for evaluating transfer from ImageNet to many other vision datasets.