mlsanityData to 14 Oct 2026

DOTA: Distributional Test-Time Adaptation of Vision-Language Models

arXiv 2409.19375 ↗First posted 28 Sep 2024cs.LG cs.AI cs.CLcited by 23

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Vision-language foundation models (VLMs), such as CLIP, exhibit remarkable performance across a wide range of tasks. However, deploying these models can be unreliable when significant distribution gaps exist between training and test data, while fine-tuning for diverse scenarios is often costly. Cache-based test-time adapters offer an efficient alternative by storing representative test samples to guide subsequent classifications. Yet, these methods typically employ naive cache management with limited capacity, leading to severe catastrophic forgetting when samples are inevitably dropped during updates. In this paper, we propose DOTA (DistributiOnal Test-time Adaptation), a simple yet effective method addressing this limitation. Crucially, instead of merely memorizing individual test samples, DOTA continuously estimates the underlying distribution of the test data stream. Test-time posterior probabilities are then computed using these dynamically estimated distributions via Bayes' theorem for adaptation. This distribution-centric approach enables the model to continually learn and adapt to the deployment environment. Extensive experiments validate that DOTA significantly mitigates forgetting and achieves state-of-the-art performance compared to existing methods.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 78 references that are in the map. First posted on arXiv 28 Sep 2024.

Builds on

50 of 78 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining30 Apr
  2. 2024Efficient Test-Time Adaptation of Vision-Language Models27 Mar
  3. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models26 Mar
  4. 2024An Empirical Study Into What Matters for Calibrating Vision-Language Models12 Feb
  5. 2024Open-Vocabulary Calibration for Fine-tuned CLIP7 Feb
  6. 2023Active Prompt Learning in Vision Language Models18 Nov
  7. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning11 Aug
  8. 2022Task Residual for Tuning Vision-Language Models18 Nov
  9. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  10. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
  11. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  12. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapters9 Oct
1 more
  1. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  7. 2013Describing Textures in the Wildcited by 2,125
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  11. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  12. 2019Natural Adversarial Examplescited by 1,408
25 more
  1. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  2. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
  3. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  4. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 610
  5. 2018Predictive Uncertainty Estimation via Prior Networkscited by 557
  6. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  7. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 377
  8. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  9. 2022Contrastive Test-Time Adaptationcited by 261
  10. 2020Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shiftcited by 236
  11. 2014Towards Open World Recognitioncited by 226
  12. 2022Parameter-free Online Test-time Adaptationcited by 189
  13. 2023Robust Test-Time Adaptation in Dynamic Scenarioscited by 189
  14. 2022Trusted Multi-View Classification with Dynamic Evidential Fusioncited by 168
  15. 2022NOTE: Robust Continual Test-time Adaptation Against Temporal Correlationcited by 151
  16. 2020Continual Prototype Evolution: Learning Online from Non-Stationary Data Streamscited by 144
  17. 2023TTN: A Domain-Shift Aware Batch Normalization in Test-Time Adaptationcited by 101
  18. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  19. 2023Universal Test-time Adaptation through Weight Ensembling, Diversity Weighting, and Prior Correctioncited by 65
  20. 2021SITA: Single Image Test-time Adaptationcited by 59
  21. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 53
  22. 2019Accurate Uncertainty Estimation and Decomposition in Ensemble Learningcited by 42
  23. 2023ID-like Prompt Learning for Few-Shot Out-of-Distribution Detectioncited by 42
  24. 2022WSSS4LUAD: Grand Challenge on Weakly-supervised Tissue Semantic Segmentation for Lung Adenocarcinomacited by 36
  25. 2024Selective Learning: Towards Robust Calibration with Dynamic Regularizationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 12.

23 papers.

Sort:

By area:

  1. 2025Bayesian Test-Time Adaptation for Vision-Language Modelscited by 22
  2. 2025COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptationcited by 13
  3. 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
  4. 2025Online Gaussian Test-Time Adaptation of Vision-Language Modelscited by 7
  5. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Surveycited by 7
  6. 2025BayesTTA: Continual-Temporal Test-Time Adaptation for Vision-Language Models via Gaussian Discriminant Analysiscited by 6
  7. 2025Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignmentcited by 6
  8. 2024Context-Aware Multimodal Pretrainingcited by 3
  9. 2026What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspectivecited by 3
  10. 2025Space Rotation with Basis Transformation for Training-free Test-Time Adaptationcited by 2
  11. 2025Bayesian Test-time Adaptation for Object Recognition and Detection with Vision-language Modelscited by 2
  12. 2026Dual-level Adaptation for Multi-Object Tracking: Building Test-Time Calibration from Experience and Intuitioncited by 2
11 more
  1. 2026Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMscited by 2
  2. 2025Unleashing the Potential of All Test Samples: Mean-Shift Guided Test-Time Adaptationcited by 1
  3. 2026Can Experts Adapt Without Training? On Test-Time Modality Generalization in MVLMscited by 1
  4. 2026Rethinking Test-Time Training: Tilting The Latent Distribution For Few-Shot Source-Free Adaptationcited by 0
  5. 2026Adapting Point Cloud Analysis via Multimodal Bayesian Distribution Learningcited by 0
  6. 2026Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibrationcited by 0
  7. 2026GRPO-TTA: Test-Time Visual Tuning for Vision-Language Models via GRPO-Driven Reinforcement Learningcited by 0
  8. 2026MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptationcited by 0
  9. 2026Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transductioncited by 0
  10. 2026Retrieval Geometry Shapes Cache-Based Clip Adaptationcited by 0
  11. 2026Not Every Correction Helps: Gain-Guided Continual Test-Time Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Prototype-Based Test-Time Adaptation of Vision-Language Models
  2. 2025Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations
  3. 2025The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
  4. 2025ETTA: Efficient Test-Time Adaptation for Vision-Language Models through Dynamic Embedding Updates
  5. 2024BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping
  6. 2025Adaptive Cache Enhancement for Test-Time Adaptation of Vision-Language Models
  7. 2024A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Models
  8. 2025Efficient Test-Time Scaling for Small Vision-Language Models