# Universal Few-shot Learning of Dense Prediction Tasks with Visual Token Matching

arXiv: https://arxiv.org/abs/2303.14969 · HTML: https://mlsanity.com/paper/2303.14969/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 49 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment](https://mlsanity.com/paper/1908.06391/) (2019)
- [Semi-Supervised Semantic Segmentation with Cross-Consistency Training](https://mlsanity.com/paper/2003.09005/) (2020)
- [Which Tasks Should Be Learned Together in Multi-task Learning?](https://mlsanity.com/paper/1905.07553/) (2019)
- [One-Shot Learning for Semantic Segmentation](https://mlsanity.com/paper/1709.03410/) (2017)
- [Frustratingly Simple Few-Shot Object Detection](https://mlsanity.com/paper/2003.06957/) (2020)
- [Rainbow Memory: Continual Learning with a Memory of Diverse Samples](https://mlsanity.com/paper/2103.17230/) (2021)
- [Few-shot Object Detection via Feature Reweighting](https://mlsanity.com/paper/1812.01866/) (2018)
- [Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector](https://mlsanity.com/paper/1908.01998/) (2019)

## Built on by (most cited first, 29 of 29)

- [Towards In-context Scene Understanding](https://mlsanity.com/paper/2306.01667/) (2023)
- [InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding](https://mlsanity.com/paper/2306.04842/) (2023)
- [Context-Aware Meta-Learning](https://mlsanity.com/paper/2310.10971/) (2023)
- [Context-Aware Interaction Network for RGB-T Semantic Segmentation](https://mlsanity.com/paper/2401.01624/) (2024)
- [Tyche: Stochastic In-Context Learning for Medical Image Segmentation](https://mlsanity.com/paper/2401.13650/) (2024)
- [Causal Unsupervised Semantic Segmentation](https://mlsanity.com/paper/2310.07379/) (2023)
- [DC-SAM: In-Context Segment Anything in Images and Videos via Dual Consistency](https://mlsanity.com/paper/2504.12080/) (2025)
- [Personalized Vision via Visual In-Context Learning](https://mlsanity.com/paper/2509.25172/) (2025)
- [Segment Together: A Versatile Paradigm for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2311.11686/) (2023)
- [Explore In-Context Segmentation via Latent Diffusion Models](https://mlsanity.com/paper/2403.09616/) (2024)
- [UniAP: Towards Universal Animal Perception in Vision via Few-shot Learning](https://mlsanity.com/paper/2308.09953/) (2023)
- [U-Netmer: U-Net meets Transformer for medical image segmentation](https://mlsanity.com/paper/2304.01401/) (2023)
- [When Multi-Task Learning Meets Partial Supervision: A Computer Vision Review](https://mlsanity.com/paper/2307.14382/) (2023)
- [Chameleon: A Data-Efficient Generalist for Dense Visual Prediction in the Wild](https://mlsanity.com/paper/2404.18459/) (2024)
- [TAM-VT: Transformation-Aware Multi-scale Video Transformer for Segmentation and Tracking](https://mlsanity.com/paper/2312.08514/) (2023)
- [Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control](https://mlsanity.com/paper/2412.12147/) (2024)
- [Group-On: Boosting One-Shot Segmentation with Supportive Query](https://mlsanity.com/paper/2404.11871/) (2024)
- [From Ideal to Real: Unified and Data-Efficient Dense Prediction for Real-World Scenarios](https://mlsanity.com/paper/2506.20279/) (2025)
- [MetaWeather: Few-Shot Weather-Degraded Image Restoration](https://mlsanity.com/paper/2308.14334/) (2023)
- [UniFS: Universal Few-shot Instance Perception with Point Representations](https://mlsanity.com/paper/2404.19401/) (2024)
- [HyperFlow: Gradient-Free Emulation of Few-Shot Fine-Tuning](https://mlsanity.com/paper/2504.15323/) (2025)
- [Universal Few-Shot Spatial Control for Diffusion Models](https://mlsanity.com/paper/2509.07530/) (2025)
- [FSDAM: Few-Shot Driving Attention Modeling via Vision-Language Coupling](https://mlsanity.com/paper/2511.12708/) (2025)
- [Enhanced Object Tracking by Self-Supervised Auxiliary Depth Estimation Learning](https://mlsanity.com/paper/2405.14195/) (2024)
- [Task-Level Contrastiveness for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2510.03509/) (2025)
- [Task-oriented Learnable Diffusion Timesteps for Universal Few-shot Learning of Dense Tasks](https://mlsanity.com/paper/2512.23210/) (2025)
- [HAD: Heterogeneity-Aware Distillation for Lifelong Heterogeneous Learning](https://mlsanity.com/paper/2603.26192/) (2026)
- [Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks](https://mlsanity.com/paper/2606.10967/) (2026)
- [Unified Video Dense Prediction from Disjoint Data](https://mlsanity.com/paper/2607.21592/) (2026)

## Abstract (arXiv)

Dense prediction tasks are a fundamental class of problems in computer vision. As supervised methods suffer from high pixel-wise labeling cost, a few-shot learning solution that can learn any dense task from a few labeled images is desired. Yet, current few-shot learning methods target a restricted set of tasks such as semantic segmentation, presumably due to challenges in designing a general and unified model that is able to flexibly and efficiently adapt to arbitrary tasks of unseen semantics. We propose Visual Token Matching (VTM), a universal few-shot learner for arbitrary dense prediction tasks. It employs non-parametric matching on patch-level embedded tokens of images and labels that encapsulates all tasks. Also, VTM flexibly adapts to any task with a tiny amount of task-specific parameters that modulate the matching algorithm. We implement VTM as a powerful hierarchical encoder-decoder architecture involving ViT backbones where token matching is performed at multiple feature hierarchies. We experiment VTM on a challenging variant of Taskonomy dataset and observe that it robustly few-shot learns various unseen dense prediction tasks. Surprisingly, it is competitive with fully supervised baselines using only 10 labeled examples of novel tasks (0.004% of full supervision) and sometimes outperforms using 0.1% of full supervision. Codes are available at https://github.com/GitGyun/visual_token_matching.