mlsanityData to 14 Oct 2026

Prompt Tuning based Adapter for Vision-Language Model Adaption

arXiv 2303.15234 ↗First posted 24 Mar 2023cs.CV cs.AIcited by 10

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large pre-trained vision-language (VL) models have shown significant promise in adapting to various downstream tasks. However, fine-tuning the entire network is challenging due to the massive number of model parameters. To address this issue, efficient adaptation methods such as prompt tuning have been proposed. We explore the idea of prompt tuning with multi-task pre-trained initialization and find it can significantly improve model performance. Based on our findings, we introduce a new model, termed Prompt-Adapter, that combines pre-trained prompt tunning with an efficient adaptation network. Our approach beat the state-of-the-art methods in few-shot image classification on the public 11 datasets, especially in settings with limited data instances such as 1 shot, 2 shots, 4 shots, and 8 shots images. Our proposed method demonstrates the promise of combining prompt tuning and parameter-efficient networks for efficient vision-language model adaptation. The code is publicly available at: https://github.com/Jingchensun/prompt_adapter.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 42 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

30 of 42 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Multitask Vision-Language Prompt Tuning21 Nov
  2. 2022Unified Vision and Language Prompt Learning13 Oct
  3. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  4. 2022Prompt Distribution Learning6 May
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  9. 2022Visual Prompt Tuningcited by 1,681
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  12. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,143
12 more
  1. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  2. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  3. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  4. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
  5. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  6. 2019A Baseline for Few-Shot Image Classificationcited by 298
  7. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  8. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  9. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  10. 2018Memory Matching Networks for One-Shot Image Recognitioncited by 65
  11. 2019Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detectioncited by 54
  12. 2022Matching Feature Sets for Few-Shot Image Classificationcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 3; 2026: 1.

10 papers.

  1. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 96
  2. 2023TCP:Textual-based Class-aware Prompt tuning for Visual-Language Modelcited by 76
  3. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  4. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuningcited by 19
  5. 2023Efficient Federated Prompt Tuning for Black-box Large Pre-trained Modelscited by 12
  6. 2024CLIP Adaptation by Intra-modal Overlap Reductioncited by 2
  7. 2024PLPP: Prompt Learning with Perplexity Is Self-Distillation for Vision-Language Modelscited by 1
  8. 2025CLAP-S: Support Set Based Adaptation for Downstream Fiber-optic Acoustic Recognitioncited by 1
  9. 2025QueryAdapter: Rapid Adaptation of Vision-Language Models in Response to Natural Language Queriescited by 1
  10. 2026Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models
  2. 2024Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning
  3. 2023Approximated Prompt Tuning for Vision-Language Pre-trained Models
  4. 2024Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves
  5. 2025Prompt Tuning Vision Language Models with Margin Regularizer for Few-Shot Learning under Distribution Shifts
  6. 2023Distribution-Aware Prompt Tuning for Vision-Language Models
  7. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment
  8. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?