# Exploring Efficient Few-shot Adaptation for Vision Transformers

arXiv: https://arxiv.org/abs/2301.02419 · HTML: https://mlsanity.com/paper/2301.02419/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 47 references that are in the map.
- First posted on arXiv 6 Jan 2023.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Meta-Learning with Latent Embedding Optimization](https://mlsanity.com/paper/1807.05960/) (2018)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [TransTrack: Multiple Object Tracking with Transformer](https://mlsanity.com/paper/2012.15460/) (2020)
- [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples](https://mlsanity.com/paper/1903.03096/) (2019)
- [Meta-Transfer Learning for Few-Shot Learning](https://mlsanity.com/paper/1812.02391/) (2018)
- [Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions](https://mlsanity.com/paper/1812.03664/) (2018)
- [Cross Attention Network for Few-shot Classification](https://mlsanity.com/paper/1910.07677/) (2019)
- [Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation](https://mlsanity.com/paper/2001.08735/) (2020)
- [CrossTransformers: spatially-aware few-shot transfer](https://mlsanity.com/paper/2007.11498/) (2020)
- [Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes](https://mlsanity.com/paper/1906.07697/) (2019)

## Built on by (most cited first, 16 of 16)

- [A Closer Look at Few-shot Classification Again](https://mlsanity.com/paper/2301.12246/) (2023)
- [Parameter-Efficient Fine-Tuning for Foundation Models](https://mlsanity.com/paper/2501.13787/) (2025)
- [Strong Baselines for Parameter Efficient Few-Shot Fine-tuning](https://mlsanity.com/paper/2304.01917/) (2023)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark](https://mlsanity.com/paper/2402.02242/) (2024)
- [Reliable Few-shot Learning under Dual Noises](https://mlsanity.com/paper/2506.16330/) (2025)
- [DETA: Denoised Task Adaptation for Few-Shot Learning](https://mlsanity.com/paper/2303.06315/) (2023)
- [When Geoscience Meets Foundation Models: Towards General Geoscience Artificial Intelligence System](https://mlsanity.com/paper/2309.06799/) (2023)
- [Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2403.04492/) (2024)
- [Neural Fine-Tuning Search for Few-Shot Learning](https://mlsanity.com/paper/2306.09295/) (2023)
- [Hint-Aug: Drawing Hints from Foundation Vision Transformers Towards Boosted Few-Shot Parameter-Efficient Tuning](https://mlsanity.com/paper/2304.12520/) (2023)
- [Split Adaptation for Pre-trained Vision Transformers](https://mlsanity.com/paper/2503.00441/) (2025)
- [Visual prompting reimagined: The power of the Activation Prompts](https://mlsanity.com/paper/2604.06440/) (2026)
- [Few-shot Classification as Multi-instance Verification: Effective Backbone-agnostic Transfer across Domains](https://mlsanity.com/paper/2507.00401/) (2025)
- [Task-Level Contrastiveness for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2510.03509/) (2025)
- [A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learning](https://mlsanity.com/paper/2609.33424/) (2026)

## Abstract (arXiv)

The task of Few-shot Learning (FSL) aims to do the inference on novel categories containing only few labeled examples, with the help of knowledge learned from base categories containing abundant labeled training samples. While there are numerous works into FSL task, Vision Transformers (ViTs) have rarely been taken as the backbone to FSL with few trials focusing on naive finetuning of whole backbone or classification layer.} Essentially, despite ViTs have been shown to enjoy comparable or even better performance on other vision tasks, it is still very nontrivial to efficiently finetune the ViTs in real-world FSL scenarios. To this end, we propose a novel efficient Transformer Tuning (eTT) method that facilitates finetuning ViTs in the FSL tasks. The key novelties come from the newly presented Attentive Prefix Tuning (APT) and Domain Residual Adapter (DRA) for the task and backbone tuning, individually. Specifically, in APT, the prefix is projected to new key and value pairs that are attached to each self-attention layer to provide the model with task-specific information. Moreover, we design the DRA in the form of learnable offset vectors to handle the potential domain gaps between base and novel data. To ensure the APT would not deviate from the initial task-specific information much, we further propose a novel prototypical regularization, which maximizes the similarity between the projected distribution of prefix and initial prototypes, regularizing the update procedure. Our method receives outstanding performance on the challenging Meta-Dataset. We conduct extensive experiments to show the efficacy of our model.