# The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models

arXiv: https://arxiv.org/abs/2506.24000 · HTML: https://mlsanity.com/paper/2506.24000/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 4 of its 77 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 4 of 4)

- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Unsupervised Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2204.03649/) (2022)
- [Beyond Model Adaptation at Test Time: A Survey](https://mlsanity.com/paper/2411.03687/) (2024)
- [Bootstrap Fine-Grained Vision-Language Alignment for Unified Zero-Shot Anomaly Localization](https://mlsanity.com/paper/2308.15939/) (2023)

## Built on by (most cited first, 8 of 8)

- [Adapting Vision-Language Models Without Labels: A Comprehensive Survey](https://mlsanity.com/paper/2508.05547/) (2025)
- [What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective](https://mlsanity.com/paper/2606.14299/) (2026)
- [Fair Context Learning for Evidence-Balanced Test-Time Adaptation in Vision-Language Models](https://mlsanity.com/paper/2602.07027/) (2026)
- [Fairness-Aware Test-Time Prompt Tuning](https://mlsanity.com/paper/2608.25707/) (2026)
- [Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation of SAM3 in Medical Image Segmentation](https://mlsanity.com/paper/2606.22963/) (2026)
- [From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model](https://mlsanity.com/paper/2608.18339/) (2026)
- [The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models](https://mlsanity.com/paper/2608.23634/) (2026)
- [On the Limits of Representation-Based Test-Time Adaptation for Multimodal LLMs](https://mlsanity.com/paper/2610.19892/) (2026)

## Abstract (arXiv)

Test-time adaptation (TTA) methods have gained significant attention for enhancing the performance of vision-language models (VLMs) such as CLIP during inference, without requiring additional labeled data. However, current TTA researches generally suffer from major limitations such as duplication of baseline results, limited evaluation metrics, inconsistent experimental settings, and insufficient analysis. These problems hinder fair comparisons between TTA methods and make it difficult to assess their practical strengths and weaknesses. To address these challenges, we introduce TTA-VLM, a comprehensive benchmark for evaluating TTA methods on VLMs. Our benchmark implements 8 episodic TTA and 7 online TTA methods within a unified and reproducible framework, and evaluates them across 15 widely used datasets. Unlike prior studies focused solely on CLIP, we extend the evaluation to SigLIP--a model trained with a Sigmoid loss--and include training-time tuning methods such as CoOp, MaPLe, and TeCoA to assess generality. Beyond classification accuracy, TTA-VLM incorporates various evaluation metrics, including robustness, calibration, out-of-distribution detection, and stability, enabling a more holistic assessment of TTA methods. Through extensive experiments, we find that 1) existing TTA methods produce limited gains compared to the previous pioneering work; 2) current TTA methods exhibit poor collaboration with training-time fine-tuning methods; 3) accuracy gains frequently come at the cost of reduced model trustworthiness. We release TTA-VLM to provide fair comparison and comprehensive evaluation of TTA methods for VLMs, and we hope it encourages the community to develop more reliable and generalizable TTA strategies.