# Learning Expressive Prompting With Residuals for Vision Transformers

arXiv: https://arxiv.org/abs/2303.15591 · HTML: https://mlsanity.com/paper/2303.15591/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 80 of its 106 references that are in the map.
- First posted on arXiv 27 Mar 2023.

## Builds on (most cited first, 30 of 80)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)

## Built on by (most cited first, 21 of 21)

- [Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model](https://mlsanity.com/paper/2404.05583/) (2024)
- [Visual Fourier Prompt Tuning](https://mlsanity.com/paper/2411.01327/) (2024)
- [Visual Instance-aware Prompt Tuning](https://mlsanity.com/paper/2507.07796/) (2025)
- [Time-, Memory- and Parameter-Efficient Visual Adaptation](https://mlsanity.com/paper/2402.02887/) (2024)
- [Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark](https://mlsanity.com/paper/2402.02242/) (2024)
- [Prompt-based Adaptation in Large-scale Vision Models: A Survey](https://mlsanity.com/paper/2510.13219/) (2025)
- [STA-VPT: SpatioTemporally Aligned Visual Prompt Tuning](https://mlsanity.com/paper/2312.10376/) (2023)
- [PRO-VPT: Distribution-Adaptive Visual Prompt Tuning via Prompt Relocation](https://mlsanity.com/paper/2503.06901/) (2025)
- [Hierarchical Side-Tuning for Vision Transformers](https://mlsanity.com/paper/2310.05393/) (2023)
- [HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalization](https://mlsanity.com/paper/2401.09716/) (2024)
- [Rethinking Layer-Wise Information Allocation for Vision Foundation Model Adaptation](https://mlsanity.com/paper/2607.21973/) (2026)
- [Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learning](https://mlsanity.com/paper/2311.13612/) (2023)
- [Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2403.08426/) (2024)
- [iVPT: Improving Task-relevant Information Sharing in Visual Prompt Tuning by Cross-layer Dynamic Connection](https://mlsanity.com/paper/2404.05207/) (2024)
- [LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning](https://mlsanity.com/paper/2402.17406/) (2024)
- [CVPT: Cross Visual Prompt Tuning](https://mlsanity.com/paper/2408.14961/) (2024)
- [Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer](https://mlsanity.com/paper/2409.13999/) (2024)
- [Layer-Specific Prompt Fusion Discovery via Differentiable Search in Vision Foundation Models](https://mlsanity.com/paper/2606.26379/) (2026)
- [pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation](https://mlsanity.com/paper/2602.22938/) (2026)
- [Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Models](https://mlsanity.com/paper/2606.19944/) (2026)
- [Adapting Vision Foundation Models with Cascaded Semantics](https://mlsanity.com/paper/2608.05393/) (2026)

## Abstract (arXiv)

Prompt learning is an efficient approach to adapt transformers by inserting learnable set of parameters into the input and intermediate representations of a pre-trained model. In this work, we present Expressive Prompts with Residuals (EXPRES) which modifies the prompt learning paradigm specifically for effective adaptation of vision transformers (ViT). Out method constructs downstream representations via learnable ``output'' tokens, that are akin to the learned class tokens of the ViT. Further for better steering of the downstream representation processed by the frozen transformer, we introduce residual learnable tokens that are added to the output of various computations. We apply EXPRES for image classification, few shot learning, and semantic segmentation, and show our method is capable of achieving state of the art prompt tuning on 3/3 categories of the VTAB benchmark. In addition to strong performance, we observe that our approach is an order of magnitude more prompt efficient than existing visual prompting baselines. We analytically show the computational benefits of our approach over weight space adaptation techniques like finetuning. Lastly we systematically corroborate the architectural design of our method via a series of ablation experiments.