mlsanityData to 14 Oct 2026

Realistic Test-Time Adaptation of Vision-Language Models

arXiv 2501.03729 ↗First posted 7 Jan 2025cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The zero-shot capabilities of Vision-Language Models (VLMs) have been widely leveraged to improve predictive performance. However, previous works on transductive or test-time adaptation (TTA) often make strong assumptions about the data distribution, such as the presence of all classes. Our work challenges these favorable deployment scenarios, and introduces a more realistic evaluation framework, including: (i) a variable number of effective classes for adaptation within a single batch, and (ii) non-i.i.d. batches of test samples in online adaptation settings. We provide comprehensive evaluations, comparisons, and ablation studies that demonstrate how current transductive or TTA methods for VLMs systematically compromise the models' initial zero-shot robustness across various realistic scenarios, favoring performance gains under advantageous assumptions about the test samples' distributions. Furthermore, we introduce StatA, a versatile method that could handle a wide range of deployment scenarios, including those with a variable number of effective classes at test time. Our approach incorporates a novel regularization term designed specifically for VLMs, which acts as a statistical anchor preserving the initial text-encoder knowledge, particularly in low-data regimes. Code available at https://github.com/MaxZanella/StatA.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 47 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

31 of 47 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024WATT: Weight Average Test-Time Adaptation of CLIP19 Jun
  2. 2024Boosting Vision-Language Models with Transduction3 Jun
  3. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models28 May
  4. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Models28 May
  5. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?3 May
  6. 2024Label Propagation for Zero-shot Classification with Vision-Language Models5 Apr
  7. 2024Efficient Test-Time Adaptation of Vision-Language Models27 Mar
  8. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models26 Mar
  9. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning11 Aug
  10. 2023Black Box Few-Shot Adaptation for Vision-Language models4 Apr
  11. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  12. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification19 Jul
1 more
  1. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2013Describing Textures in the Wildcited by 2,125
  3. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  5. 2022MaPLe: Multi-modal Prompt Learningcited by 687
  6. 2022Contrastive Test-Time Adaptationcited by 261
  7. 2018Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learningcited by 211
  8. 2022Parameter-free Online Test-time Adaptationcited by 189
  9. 2023Robust Test-Time Adaptation in Dynamic Scenarioscited by 189
  10. 2022NOTE: Robust Continual Test-time Adaptation Against Temporal Correlationcited by 151
  11. 2020Laplacian Regularized Few-Shot Learningcited by 92
  12. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 53
6 more
  1. 2024Transductive Zero-Shot and Few-Shot CLIPcited by 32
  2. 2013An Information-Theoretic Analysis of Hard and Soft Assignment Methods for Clusteringcited by 17
  3. 2022Towards Practical Few-Shot Query Sets: Transductive Minimum Description Length Inferencecited by 9
  4. 2024Enhancing Remote Sensing Vision-Language Models for Zero-Shot Scene Classificationcited by 8
  5. 2024Boosting Vision-Language Models for Histopathology Classification: Predict all at oncecited by 8
  6. 2015Volumetric Bias in Segmentation and Reconstruction: Secrets and Solutionscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2025Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignmentcited by 6
  2. 2026ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learningcited by 4
  3. 2026What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspectivecited by 3
  4. 2025Uniformity First: Uniformity-aware Test-time Adaptation of Vision-language Models against Image Corruptioncited by 1
  5. 2025CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Trainingcited by 1
  6. 2026ORION: ORthonormal Text Encoding for Universal VLM AdaptatIONcited by 1
  7. 2026ProtoDCS: Towards Robust and Efficient Open-Set Test-Time Adaptation for Vision-Language Modelscited by 1
  8. 2026Adapting Point Cloud Analysis via Multimodal Bayesian Distribution Learningcited by 0
  9. 2026Ramen: Robust Test-Time Adaptation of Vision-Language Models with Active Sample Selectioncited by 0
  10. 2026Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Modelscited by 0
  11. 2026When Can Test-Time Adaptation Help Zero-Shot CT Vision-Language Models?cited by 0
  12. 2026Continual Visual Learning under Evolving Semantic Concept Shiftcited by 0
2 more
  1. 2026Towards Purified Multi-Label Test-Time Adaptation of Vision-Language Modelscited by 0
  2. 2026Not Every Correction Helps: Gain-Guided Continual Test-Time Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
  2. 2024Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models
  3. 2026Prototype-Based Test-Time Adaptation of Vision-Language Models
  4. 2026Efficient Test-time Adaptation through Candidate Verification and Divergence Shifts
  5. 2025Efficient Test-Time Scaling for Small Vision-Language Models
  6. 2025Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations
  7. 2025Ultra-Light Test-Time Adaptation for Vision--Language Models
  8. 2025Adaptive Cache Enhancement for Test-Time Adaptation of Vision-Language Models