mlsanityData to 7 Oct 2026

Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning

arXiv 2408.16486 ↗First posted 29 Aug 2024cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Adapting pre-trained models to open classes is a challenging problem in machine learning. Vision-language models fully explore the knowledge of text modality, demonstrating strong zero-shot recognition performance, which is naturally suited for various open-set problems. More recently, some research focuses on fine-tuning such models to downstream tasks. Prompt tuning methods achieved huge improvements by learning context vectors on few-shot data. However, through the evaluation under open-set adaptation setting with the test data including new classes, we find that there exists a dilemma that learned prompts have worse generalization abilities than hand-crafted prompts. In this paper, we consider combining the advantages of both and come up with a test-time prompt tuning approach, which leverages the maximum concept matching (MCM) scores as dynamic weights to generate an input-conditioned prompt for each image during test. Through extensive experiments on 11 different datasets, we show that our proposed method outperforms all comparison methods on average considering both base and new classes. The code is available at https://github.com/gaozhengqing/TTPT

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 43 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

34 of 43 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Improving Zero-Shot Generalization for CLIP with Synthesized Prompts14 Jul
  2. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?9 Jun
  3. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  4. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov
  5. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2013Describing Textures in the Wildcited by 2,110
  12. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
16 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  3. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  4. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  5. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  6. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  7. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  8. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  9. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  10. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  11. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  12. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  13. 2022Delving into Out-of-Distribution Detection with Vision-Language Representationscited by 215
  14. 2021Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIPcited by 151
  15. 2023LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learningcited by 107
  16. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
  2. 2024Can Better Text Semantics in Prompt Tuning Improve VLM Generalization?
  3. 2024In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Model
  4. 2025Prompting without Panic: Attribute-aware, Zero-shot, Test-Time Calibration
  5. 2023Prompt Tuning based Adapter for Vision-Language Model Adaption
  6. 2024Generalizable Prompt Tuning for Vision-Language Models
  7. 2025A Closer Look at Conditional Prompt Tuning for Vision-Language Models
  8. 2024Tuning Vision-Language Models with Candidate Labels by Prompt Alignment