# Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages

arXiv: https://arxiv.org/abs/2503.11609 · HTML: https://mlsanity.com/paper/2503.11609/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 6 of its 53 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 6 of 6)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.15647/) (2023)
- [Early Stopping without a Validation Set](https://mlsanity.com/paper/1703.09580/) (2017)
- [Diversified in-domain synthesis with efficient fine-tuning for few-shot classification](https://mlsanity.com/paper/2312.03046/) (2023)

## Built on by (most cited first, 17 of 17)

- [MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptation](https://mlsanity.com/paper/2602.21397/) (2026)
- [Optimization-Inspired Few-Shot Adaptation for Large Language Models](https://mlsanity.com/paper/2505.19107/) (2025)
- [VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2511.22664/) (2025)
- [Spotlighter: Revisiting Prompt Tuning from a Representative Mining View](https://mlsanity.com/paper/2509.00905/) (2025)
- [CLIPoint3D: Language-Grounded Few-Shot Unsupervised 3D Point Cloud Domain Adaptation](https://mlsanity.com/paper/2602.20409/) (2026)
- [Reevaluating the Intra-Modal Misalignment Hypothesis in CLIP](https://mlsanity.com/paper/2603.16100/) (2026)
- [Robust Prompt Tuning for Vision-Language Models with Mild Semantic Noise](https://mlsanity.com/paper/2508.04677/) (2025)
- [Provably Robust Adaptation for Language-Empowered Foundation Models](https://mlsanity.com/paper/2510.08659/) (2025)
- [RMAdapter: Reconstruction-based Multi-Modal Adapter for Vision-Language Models](https://mlsanity.com/paper/2512.06811/) (2025)
- [Auxiliary Descriptive Knowledge for Few-Shot Adaptation of Vision-Language Model](https://mlsanity.com/paper/2512.17313/) (2025)
- [CAPT: Confusion-Aware Prompt Tuning for Reducing Vision-Language Misalignment](https://mlsanity.com/paper/2603.02557/) (2026)
- [Training-Only Heterogeneous Image-Patch-Text Graph Supervision for Advancing Few-Shot Learning Adapters](https://mlsanity.com/paper/2603.18101/) (2026)
- [STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition](https://mlsanity.com/paper/2605.13202/) (2026)
- [PrAda: Few-Shot Visual Adaptation for Text-Prompted Segmentation](https://mlsanity.com/paper/2605.19623/) (2026)
- [Decompose, Compare, and Decide: Multimodal LLMs are Implicit Few-Shot Learners](https://mlsanity.com/paper/2607.00125/) (2026)
- [ProCAP: Probabilistic Cross-Attentive Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2609.30434/) (2026)
- [AdaKerNet: Neural Kernel Decoding for Task-Adaptive Prediction with Multimodal Large Models](https://mlsanity.com/paper/2609.36368/) (2026)

## Abstract (arXiv)

An old-school recipe for training a classifier is to (i) learn a good feature extractor and (ii) optimize a linear layer atop. When only a handful of samples are available per category, as in Few-Shot Adaptation (FSA), data are insufficient to fit a large number of parameters, rendering the above impractical. This is especially true with large pre-trained Vision-Language Models (VLMs), which motivated successful research at the intersection of Parameter-Efficient Fine-tuning (PEFT) and FSA. In this work, we start by analyzing the learning dynamics of PEFT techniques when trained on few-shot data from only a subset of categories, referred to as the ``base'' classes. We show that such dynamics naturally splits into two distinct phases: (i) task-level feature extraction and (ii) specialization to the available concepts. To accommodate this dynamic, we then depart from prompt- or adapter-based methods and tackle FSA differently. Specifically, given a fixed computational budget, we split it to (i) learn a task-specific feature extractor via PEFT and (ii) train a linear classifier on top. We call this scheme Two-Stage Few-Shot Adaptation (2SFS). Differently from established methods, our scheme enables a novel form of selective inference at a category level, i.e., at test time, only novel categories are embedded by the adapted text encoder, while embeddings of base categories are available within the classifier. Results with fixed hyperparameters across two settings, three backbones, and eleven datasets, show that 2SFS matches or surpasses the state-of-the-art, while established methods degrade significantly across settings.