mlsanityData to 7 Oct 2026

When Can Test-Time Adaptation Help Zero-Shot CT Vision-Language Models?

arXiv 2607.15556 ↗First posted 17 Jul 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

3D CT vision-language models (VLMs) classify abnormalities from text prompts in a zero-shot manner, enabling cross-institution deployment where labels are scarce and clinical tasks shift faster than supervised models can be retrained. A real CT scan, however, typically contains several co-occurring abnormalities, and the reliability of zero-shot multi-label prediction under distribution shift remains poorly understood. Test-time adaptation (TTA) updates a model on unlabeled target scans without source data or target annotations, yet existing TTA methods target multi-class softmax prediction on natural images or 2D medical segmentation, and none addresses unsupervised multi-label adaptation for zero-shot 3D CT VLMs. We study when TTA helps zero-shot 3D CT VLMs. A controlled diagnostic analysis shows that TTA is conditional: the volumetric input must preserve the encoder's depth structure, and the base representation must transfer to the target cohort, with depth reduction alone lowering internal AUROC by more than 0.12. We then focus on the regime where the base model already separates present from absent abnormalities. We introduce CARVE (Cardinality-Aware Retained-View Entropy), the first TTA method for this setting. CARVE estimates a sample-specific positive-label cardinality hatk, optimizes a top-hatk objective to preserve co-occurring abnormalities, and performs memory-efficient multi-view adaptation by scoring weak 3D views without gradients before updating on a retained subset. Across contrastive CT-CLIP and anatomy-aware fVLM, CARVE provides the most consistent improvements across multi-label, three-class, and binary CT tasks when the base model is already discriminative. These results establish multi-label TTA for zero-shot 3D CT VLMs as a distinct problem and CARVE as a cardinality-aware solution.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 24 references that are in the map. First posted on arXiv 17 Jul 2026.

Builds on

19 of 24 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding24 Jan
  2. 2024Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset10 Jun

Other references · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  3. 2022Continual Test-Time Domain Adaptationcited by 520
  4. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  5. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 409
  6. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  7. 2024Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomographycited by 192
  8. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuningcited by 150
  9. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 138
  10. 2023Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Modelscited by 37
  11. 2024CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarioscited by 29
  12. 2025Bayesian Test-Time Adaptation for Vision-Language Modelscited by 21
5 more
  1. 2022Test-time Adaptation with Calibration of Medical Image Classification Nets for Label Distribution Shiftcited by 20
  2. 2025Realistic Test-Time Adaptation of Vision-Language Modelscited by 14
  3. 2024Gradient Alignment Improves Test-Time Adaptation for Medical Image Segmentationcited by 11
  4. 2025Multi-Label Test-Time Adaptation with Bound Entropy Minimizationcited by 10
  5. 2025A Large Scale Benchmark for Test Time Adaptation Methods in Medical Image Segmentationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TCLA: Training-Free Class-wise Logit Adaptation for Medical Vision-Language Models
  2. 2025Scaling Down to Scale Up: Towards Operationally-Efficient and Deployable Clinical Models via Cross-Modal Low-Rank Adaptation for Medical Vision-Language Models
  3. 2025Unified Supervision For Vision-Language Modeling in 3D Computed Tomography
  4. 2025Noisy Test-Time Adaptation in Vision-Language Models
  5. 2025Few-Shot, Now for Real: Medical VLMs Adaptation without Balanced Sets or Validation
  6. 2025T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysis
  7. 2025Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis
  8. 2025Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation