mlsanityData to 7 Oct 2026

Efficient Test-Time Adaptation of Vision-Language Models

arXiv 2403.18293 ↗First posted 27 Mar 2024cs.CVcited by 138

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

also in 1 more
Abstract (arXiv)

Test-time adaptation with pre-trained vision-language models has attracted increasing attention for tackling distribution shifts during the test time. Though prior studies have achieved very promising performance, they involve intensive computation which is severely unaligned with test-time adaptation. We design TDA, a training-free dynamic adapter that enables effective and efficient test-time adaptation with vision-language models. TDA works with a lightweight key-value cache that maintains a dynamic queue with few-shot pseudo labels as values and the corresponding test-sample features as keys. Leveraging the key-value cache, TDA allows adapting to test data gradually via progressive pseudo label refinement which is super-efficient without incurring any backpropagation. In addition, we introduce negative pseudo labeling that alleviates the adverse impact of pseudo label noises by assigning pseudo labels to certain negative classes when the model is uncertain about its pseudo label predictions. Extensive experiments over two benchmarks demonstrate TDA's superior effectiveness and efficiency as compared with the state-of-the-art. The code has been released in urlhttps://kdiaaa.github.io/tda/.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 52 references that are in the map. First posted on arXiv 27 Mar 2024.

Builds on

39 of 52 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning11 Aug
  2. 2022Task Residual for Tuning Vision-Language Models18 Nov
  3. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  4. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
  7. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2017Billion-scale similarity search with GPUscited by 2,477
  8. 2013Describing Textures in the Wildcited by 2,110
  9. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  10. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
20 more
  1. 2019Natural Adversarial Examplescited by 1,401
  2. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  3. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  4. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  5. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  6. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  7. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  8. 2022Contrastive Test-Time Adaptationcited by 258
  9. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  10. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  11. 2022Parameter-free Online Test-time Adaptationcited by 184
  12. 2020Adaptive Risk Minimization: Learning to Adapt to Domain Shiftcited by 183
  13. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  14. 2019NLNL: Negative Learning for Noisy Labelscited by 151
  15. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  16. 2017Unbounded cache model for online language modeling with open vocabularycited by 37
  17. 2020Test-time Unsupervised Domain Adaptationcited by 37
  18. 2021Joint Negative and Positive Learning for Noisy Labelscited by 34
  19. 2018A Simple Cache Model for Image Recognitioncited by 20
  20. 2023Temporal Coherent Test-Time Optimization for Robust Video Classificationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 66; 2026: 54.

138 papers.

Sort:

By area:

  1. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Modelscited by 36
  2. 2024MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoningcited by 35
  3. 2024WATT: Weight Average Test-Time Adaptation of CLIPcited by 33
  4. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  5. 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
  6. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  7. 2024DOTA: Distributional Test-Time Adaptation of Vision-Language Modelscited by 22
  8. 2024Historical Test-time Prompt Tuning for Vision Foundation Modelscited by 21
  9. 2025Bayesian Test-Time Adaptation for Vision-Language Modelscited by 21
  10. 2024BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrappingcited by 17
  11. 2024CLIPArTT: Adaptation of CLIP to New Domains at Test Timecited by 14
  12. 2025Realistic Test-Time Adaptation of Vision-Language Modelscited by 14
38 more
  1. 2025COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptationcited by 13
  2. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  3. 2024textttBATCLIP: Bimodal Online Test-Time Adaptation for CLIPcited by 11
  4. 2025SCAP: Transductive Test-Time Adaptation via Supportive Clique-based Attribute Promptingcited by 11
  5. 2025Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptationcited by 11
  6. 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Modelcited by 10
  7. 2025Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variationscited by 9
  8. 2025Latte: Collaborative Test-Time Adaptation of Vision-Language Models in Federated Learningcited by 9
  9. 2024Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Modelcited by 8
  10. 2025Noisy Test-Time Adaptation in Vision-Language Modelscited by 8
  11. 2025Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EMcited by 8
  12. 2025LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Modelscited by 7
  13. 2025Test-Time Visual In-Context Tuningcited by 7
  14. 2026Steer2Adapt: Dynamically Composing Steering Vectors Elicits Efficient Adaptation of LLMscited by 7
  15. 2025Online Gaussian Test-Time Adaptation of Vision-Language Modelscited by 6
  16. 2025Bidirectional Prototype-Reward co-Evolution for Test-Time Adaptation of Vision-Language Modelscited by 6
  17. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  18. 2025Multi-Cache Enhanced Prototype Learning for Test-Time Generalization of Vision-Language Modelscited by 6
  19. 2024On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Surveycited by 5
  20. 2024Is Less More? Exploring Token Condensation as Training-free Test-time Adaptationcited by 5
  21. 2025Noise is an Efficient Learner for Zero-Shot Vision-Language Modelscited by 5
  22. 2025From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selectioncited by 5
  23. 2025Test-Time Distillation for Continual Model Adaptationcited by 5
  24. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  25. 2025Test-Time Consistency in Vision Language Modelscited by 5
  26. 2025CLIPTTA: Robust Contrastive Vision-Language Test-Time Adaptationcited by 5
  27. 2025Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignmentcited by 5
  28. 2025Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptionscited by 5
  29. 2026Test-Time Adaptation via Cache Personalization for Facial Expression Recognition in Videoscited by 5
  30. 2025Point-Cache: Test-time Dynamic and Hierarchical Cache for Robust and Generalizable Point Cloud Analysiscited by 4
  31. 2025Negation-Aware Test-Time Adaptation for Vision-Language Modelscited by 4
  32. 2025Test-Time Adaptive Object Detection with Foundation Modelcited by 4
  33. 2024Efficient Open Set Single Image Test Time Adaptation of Vision Language Modelscited by 3
  34. 2025To Trust Or Not To Trust Your Vision-Language Model's Predictioncited by 3
  35. 2025ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updatescited by 3
  36. 2025Adaptive Cache Enhancement for Test-Time Adaptation of Vision-Language Modelscited by 3
  37. 2025An Evidence-Based Post-Hoc Adjustment Framework for Anomaly Detection Under Data Contaminationcited by 3
  38. 2025Boosting Skeleton-based Zero-Shot Action Recognition with Training-Free Test-Time Adaptationcited by 3

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Prototype-Based Test-Time Adaptation of Vision-Language Models
  2. 2025The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
  3. 2026Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts
  4. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Models
  5. 2025Efficient Test-Time Scaling for Small Vision-Language Models
  6. 2024Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models
  7. 2025Ultra-Light Test-Time Adaptation for Vision--Language Models
  8. 2026To Adapt or Not to Adapt? Selective Adaptation for Vision-Language Models