mlsanityData to 7 Oct 2026

On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?

arXiv 2405.02266 ↗First posted 3 May 2024cs.CVcited by 65

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The development of large vision-language models, notably CLIP, has catalyzed research into effective adaptation techniques, with a particular focus on soft prompt tuning. Conjointly, test-time augmentation, which utilizes multiple augmented views of a single image to enhance zero-shot generalization, is emerging as a significant area of interest. This has predominantly directed research efforts toward test-time prompt tuning. In contrast, we introduce a robust MeanShift for Test-time Augmentation (MTA), which surpasses prompt-based methods without requiring this intensive training procedure. This positions MTA as an ideal solution for both standalone and API-based applications. Additionally, our method does not rely on ad hoc rules (e.g., confidence threshold) used in some previous test-time augmentation techniques to filter the augmented views. Instead, MTA incorporates a quality assessment variable for each view directly into its optimization process, termed as the inlierness score. This score is jointly optimized with a density mode seeking process, leading to an efficient training- and hyperparameter-free approach. We extensively benchmark our method on 15 datasets and demonstrate MTA's superiority and computational efficiency. Deployed easily as plug-and-play module on top of zero-shot models and state-of-the-art few-shot methods, MTA shows systematic and consistent improvements.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 68 references that are in the map. First posted on arXiv 3 May 2024.

Builds on

43 of 68 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning11 Aug
  2. 2023Black Box Few-Shot Adaptation for Vision-Language models4 Apr
  3. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimization23 Mar
  4. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Models28 Nov
  5. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  6. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  7. 2022Prompt-aligned Gradient for Prompt Tuning30 May
  8. 2022Prompt Distribution Learning6 May
  9. 2022Unsupervised Prompt Learning for Vision-Language Models7 Apr
  10. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  11. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  12. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  12. 2019Natural Adversarial Examplescited by 1,401
19 more
  1. 2021Grounded Language-Image Pre-trainingcited by 1,147
  2. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  3. 2019How Can We Know What Language Models Know?cited by 953
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  7. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  8. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  9. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  10. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  11. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  12. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
  13. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  14. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  15. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  16. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  17. 2022LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Modelscited by 58
  18. 2022Improving Zero-shot Generalization and Robustness of Multi-modal Modelscited by 44
  19. 2023Transductive Learning for Textual Few-Shot Classification in API-based Embedding Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 33; 2026: 22.

65 papers.

Sort:

By area:

  1. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  2. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Modelscited by 36
  3. 2024Boosting Vision-Language Models with Transductioncited by 36
  4. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Modelscited by 36
  5. 2025R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuningcited by 33
  6. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  7. 2024TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Modelscited by 22
  8. 2025Bayesian Test-Time Adaptation for Vision-Language Modelscited by 21
  9. 2025A Survey on Efficient Vision-Language Modelscited by 19
  10. 2025Realistic Test-Time Adaptation of Vision-Language Modelscited by 14
  11. 2025SCAP: Transductive Test-Time Adaptation via Supportive Clique-based Attribute Promptingcited by 11
  12. 2024Enhancing Remote Sensing Vision-Language Models for Zero-Shot Scene Classificationcited by 8
38 more
  1. 2024Boosting Vision-Language Models for Histopathology Classification: Predict all at oncecited by 8
  2. 2025Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EMcited by 8
  3. 2025LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Modelscited by 7
  4. 2025Test-Time Visual In-Context Tuningcited by 7
  5. 2025Online Gaussian Test-Time Adaptation of Vision-Language Modelscited by 6
  6. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  7. 2024On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Surveycited by 5
  8. 2024Is Less More? Exploring Token Condensation as Training-free Test-time Adaptationcited by 5
  9. 2025Noise is an Efficient Learner for Zero-Shot Vision-Language Modelscited by 5
  10. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  11. 2025Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignmentcited by 5
  12. 2025Point-Cache: Test-time Dynamic and Hierarchical Cache for Robust and Generalizable Point Cloud Analysiscited by 4
  13. 2025ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updatescited by 3
  14. 2025Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Modelscited by 3
  15. 2025VTD-CLIP: Video-to-Text Discretization via Prompting CLIPcited by 2
  16. 2025One Last Attention for Your Vision-Language Modelcited by 2
  17. 2025T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysiscited by 2
  18. 2025TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Modelscited by 2
  19. 2026ClipTTT: CLIP-Guided Test-Time Training Helps LVLMs See Bettercited by 2
  20. 2026What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspectivecited by 2
  21. 2024Physically Interpretable Probabilistic Domain Characterizationcited by 1
  22. 2025Flatness Guided Test-Time Adaptation for Vision-Language Modelscited by 1
  23. 2025Space Rotation with Basis Transformation for Training-free Test-Time Adaptationcited by 1
  24. 2025Enhancing Zero-Shot Image Recognition in Vision-Language Models through Human-like Concept Guidancecited by 1
  25. 2025Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruptioncited by 1
  26. 2025Unleashing the Potential of All Test Samples: Mean-Shift Guided Test-Time Adaptationcited by 1
  27. 2025Training-Free Label Space Alignment for Universal Domain Adaptationcited by 1
  28. 2025EMO-TTA: Improving Test-Time Adaptation of Audio-Language Models for Speech Emotion Recognitioncited by 1
  29. 2025Bayesian Test-time Adaptation for Object Recognition and Detection with Vision-language Modelscited by 1
  30. 2026ORION: ORthonormal Text Encoding for Universal VLM AdaptatIONcited by 1
  31. 2026ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Modelscited by 1
  32. 2026A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Modelscited by 1
  33. 2026When CLIP Sees More, It Fights Back Harder: Multi-View Guided Adaptive Counterattacks for Test-Time Adversarial Robustnesscited by 1
  34. 2025Conformal Predictions for Human Action Recognition with Vision-Language Modelscited by 0
  35. 2025Dynamic Rank Adaptation for Vision-Language Modelscited by 0
  36. 2025Language-Aware Information Maximization for Transductive Few-Shot CLIPcited by 0
  37. 2025Measurement Plasticity: Sensor-Level Adaptation for Vision-Language Modelscited by 0
  38. 2026Fair Context Learning for Evidence-Balanced Test-Time Adaptation in Vision-Language Modelscited by 0

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MetaTPT: Meta Test-time Prompt Tuning for Vision-Language Models
  2. 2026ZOMP: Zeroth-Order Multi-Modal Prompt Tuning for Vision-Language Models
  3. 2025Efficient Test-Time Scaling for Small Vision-Language Models
  4. 2024Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models
  5. 2025MINT: Memory-Infused Prompt Tuning at Test-time for CLIP
  6. 2025Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models
  7. 2023Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models
  8. 2026Robustifying Vision-Language Models via Test-Time Prompt Adaptation