# DOTA: Distributional Test-Time Adaptation of Vision-Language Models

arXiv: https://arxiv.org/abs/2409.19375 · HTML: https://mlsanity.com/paper/2409.19375/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 50 of its 78 references that are in the map.
- First posted on arXiv 28 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [MEMO: Test Time Robustness via Adaptation and Augmentation](https://mlsanity.com/paper/2110.09506/) (2021)
- [Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification](https://mlsanity.com/paper/2207.09519/) (2022)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [Contrastive Test-Time Adaptation](https://mlsanity.com/paper/2204.10377/) (2022)
- [Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift](https://mlsanity.com/paper/2006.10963/) (2020)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [Parameter-free Online Test-time Adaptation](https://mlsanity.com/paper/2201.05718/) (2022)
- [Robust Test-Time Adaptation in Dynamic Scenarios](https://mlsanity.com/paper/2303.13899/) (2023)

## Built on by (most cited first, 23 of 23)

- [Bayesian Test-Time Adaptation for Vision-Language Models](https://mlsanity.com/paper/2503.09248/) (2025)
- [COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptation](https://mlsanity.com/paper/2503.23388/) (2025)
- [Dynamic Multimodal Prototype Learning in Vision-Language Models](https://mlsanity.com/paper/2507.03657/) (2025)
- [Online Gaussian Test-Time Adaptation of Vision-Language Models](https://mlsanity.com/paper/2501.04352/) (2025)
- [Adapting Vision-Language Models Without Labels: A Comprehensive Survey](https://mlsanity.com/paper/2508.05547/) (2025)
- [BayesTTA: Continual-Temporal Test-Time Adaptation for Vision-Language Models via Gaussian Discriminant Analysis](https://mlsanity.com/paper/2507.08607/) (2025)
- [Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment](https://mlsanity.com/paper/2508.15568/) (2025)
- [Context-Aware Multimodal Pretraining](https://mlsanity.com/paper/2411.15099/) (2024)
- [What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective](https://mlsanity.com/paper/2606.14299/) (2026)
- [Space Rotation with Basis Transformation for Training-free Test-Time Adaptation](https://mlsanity.com/paper/2502.19946/) (2025)
- [Bayesian Test-time Adaptation for Object Recognition and Detection with Vision-language Models](https://mlsanity.com/paper/2510.02750/) (2025)
- [Dual-level Adaptation for Multi-Object Tracking: Building Test-Time Calibration from Experience and Intuition](https://mlsanity.com/paper/2603.21629/) (2026)
- [Dual Distribution Estimation for Zero-shot Noisy Test-Time Adaptation with VLMs](https://mlsanity.com/paper/2606.25758/) (2026)
- [Unleashing the Potential of All Test Samples: Mean-Shift Guided Test-Time Adaptation](https://mlsanity.com/paper/2507.00462/) (2025)
- [Can Experts Adapt Without Training? On Test-Time Modality Generalization in MVLMs](https://mlsanity.com/paper/2607.16726/) (2026)
- [Rethinking Test-Time Training: Tilting The Latent Distribution For Few-Shot Source-Free Adaptation](https://mlsanity.com/paper/2602.02633/) (2026)
- [Adapting Point Cloud Analysis via Multimodal Bayesian Distribution Learning](https://mlsanity.com/paper/2603.22070/) (2026)
- [Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration](https://mlsanity.com/paper/2604.19093/) (2026)
- [GRPO-TTA: Test-Time Visual Tuning for Vision-Language Models via GRPO-Driven Reinforcement Learning](https://mlsanity.com/paper/2605.03403/) (2026)
- [MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation](https://mlsanity.com/paper/2605.17743/) (2026)
- [Von Mises-Fisher Mixture Model with Dynamic Shrinkage for Realistic Test-Time Transduction](https://mlsanity.com/paper/2607.15851/) (2026)
- [Retrieval Geometry Shapes Cache-Based Clip Adaptation](https://mlsanity.com/paper/2609.23409/) (2026)
- [Not Every Correction Helps: Gain-Guided Continual Test-Time Adaptation](https://mlsanity.com/paper/2609.36655/) (2026)

## Abstract (arXiv)

Vision-language foundation models (VLMs), such as CLIP, exhibit remarkable performance across a wide range of tasks. However, deploying these models can be unreliable when significant distribution gaps exist between training and test data, while fine-tuning for diverse scenarios is often costly. Cache-based test-time adapters offer an efficient alternative by storing representative test samples to guide subsequent classifications. Yet, these methods typically employ naive cache management with limited capacity, leading to severe catastrophic forgetting when samples are inevitably dropped during updates. In this paper, we propose DOTA (DistributiOnal Test-time Adaptation), a simple yet effective method addressing this limitation. Crucially, instead of merely memorizing individual test samples, DOTA continuously estimates the underlying distribution of the test data stream. Test-time posterior probabilities are then computed using these dynamically estimated distributions via Bayes' theorem for adaptation. This distribution-centric approach enables the model to continually learn and adapt to the deployment environment. Extensive experiments validate that DOTA significantly mitigates forgetting and achieves state-of-the-art performance compared to existing methods.