mlsanityData to 7 Oct 2026

A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Models

arXiv 2405.14977 ↗First posted 23 May 2024cs.CVcited by 10

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

In deep learning, maintaining model robustness against distribution shifts is critical. This work explores a broad range of possibilities to adapt vision-language foundation models at test-time, with a particular emphasis on CLIP and its variants. The study systematically examines prompt-based techniques and existing test-time adaptation methods, aiming to improve the robustness under distribution shift in diverse real-world scenarios. Specifically, the investigation covers various prompt engineering strategies, including handcrafted prompts, prompt ensembles, and prompt learning techniques. Additionally, we introduce a vision-text-space ensemble that substantially enhances average performance compared to text-space-only ensembles. Since online test-time adaptation has shown to be effective to mitigate performance drops under distribution shift, the study extends its scope to evaluate the effectiveness of existing test-time adaptation methods that were originally designed for vision-only classification models. Through extensive experimental evaluations conducted across multiple datasets and diverse model architectures, the research demonstrates the effectiveness of these adaptation strategies. Code is available at: https://github.com/mariodoebler/test-time-adaptation

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 53 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

41 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning11 Aug
  2. 2023ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation4 Aug
  3. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  4. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2013Describing Textures in the Wildcited by 2,110
25 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  3. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  5. 2019Natural Adversarial Examplescited by 1,401
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  7. 2020Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptationcited by 844
  8. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  11. 2022Continual Test-Time Domain Adaptationcited by 520
  12. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  13. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  14. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  15. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 263
  16. 2022Contrastive Test-Time Adaptationcited by 258
  17. 2023Robust Test-Time Adaptation in Dynamic Scenarioscited by 185
  18. 2022Parameter-free Online Test-time Adaptationcited by 184
  19. 2022NOTE: Robust Continual Test-time Adaptation Against Temporal Correlationcited by 150
  20. 2024Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factorscited by 106
  21. 2022Robust Mean Teacher for Continual and Gradual Test-Time Adaptationcited by 102
  22. 2021Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformationcited by 67
  23. 2023Universal Test-time Adaptation through Weight Ensembling, Diversity Weighting, and Prior Correctioncited by 65
  24. 2022Introducing Intermediate Domains for Effective Self-Training during Test-Timecited by 12
  25. 2024Diversity-aware Buffer for Coping with Temporally Correlated Data Streams in Online Test-time Adaptationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 4.

10 papers.

  1. 2024textttBATCLIP: Bimodal Online Test-Time Adaptation for CLIPcited by 11
  2. 2025Latte: Collaborative Test-Time Adaptation of Vision-Language Models in Federated Learningcited by 9
  3. 2024Is Less More? Exploring Token Condensation as Training-free Test-time Adaptationcited by 5
  4. 2025Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptionscited by 5
  5. 2024Semantic Anchor Transport: Robust Test-Time Adaptation for Vision-Language Modelscited by 3
  6. 2025textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Timecited by 2
  7. 2026Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistencycited by 2
  8. 2026Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selectioncited by 0
  9. 2026Suppress and Diversify: Refining Robust Pathways for Corruption Robustnesscited by 0
  10. 2026From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates
  2. 2025The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
  3. 2025AmPLe: Supporting Vision-Language Models via Adaptive-Debiased Ensemble Multi-Prompt Learning
  4. 2024DOTA: Distributional Test-Time Adaptation of Vision-Language Models
  5. 2024In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-language Model
  6. 2025MINT: Memory-Infused Prompt Tuning at Test-time for CLIP
  7. 2024Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning
  8. 2025Realistic Test-Time Adaptation of Vision-Language Models