mlsanityData to 7 Oct 2026

Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models

arXiv 2407.15913 ↗First posted 22 Jul 2024cs.CVcited by 17

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

The conventional modus operandi for adapting pre-trained vision-language models (VLMs) during test-time involves tuning learnable prompts, ie, test-time prompt tuning. This paper introduces Test-Time Low-rank adaptation (TTL) as an alternative to prompt tuning for zero-shot generalization of large-scale VLMs. Taking inspiration from recent advancements in efficiently fine-tuning large language models, TTL offers a test-time parameter-efficient adaptation approach that updates the attention weights of the transformer encoder by maximizing prediction confidence. The self-supervised confidence maximization objective is specified using a weighted entropy loss that enforces consistency among predictions of augmented samples. TTL introduces only a small amount of trainable parameters for low-rank adapters in the model space while keeping the prompts and backbone frozen. Extensive experiments on a variety of natural distribution and cross-domain tasks show that TTL can outperform other techniques for test-time optimization of VLMs in strict zero-shot settings. Specifically, TTL outperforms test-time prompt tuning baselines with a significant improvement on average. Our code is available at at https://github.com/Razaimam45/TTL-Test-Time-Low-Rank-Adaptation.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 57 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

39 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Learning to Prompt with Text Only Supervision for Vision-Language Models4 Jan
  2. 2023Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization2 Nov
  3. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuning11 Aug
  4. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
  5. 2022Conditional Prompt Learning for Vision-Language Models10 Mar
  6. 2021Learning to Prompt for Vision-Language Models2 Sep

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  10. 2022Visual Prompt Tuningcited by 1,674
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
21 more
  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  2. 2019Natural Adversarial Examplescited by 1,401
  3. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  4. 2021Robust fine-tuning of zero-shot modelscited by 785
  5. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  6. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  7. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  8. 2018Recognition in Terra Incognitacited by 655
  9. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  10. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  11. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 467
  12. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  13. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  14. 2023Self-regulating Prompts: Foundational Model Adaptation without Forgettingcited by 272
  15. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  16. 2024Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factorscited by 106
  17. 2022CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attentioncited by 103
  18. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  19. 2021Examining and Combating Spurious Features under Distribution Shiftcited by 64
  20. 2023Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentationcited by 61
  21. 2023Going Beyond Nouns With Vision & Language Models Using Synthetic Datacited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 7.

17 papers.

Sort:

By area:

  1. 2025On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?cited by 10
  2. 2025Generalizing vision-language models to novel domains: A comprehensive surveycited by 6
  3. 2025Noise is an Efficient Learner for Zero-Shot Vision-Language Modelscited by 5
  4. 2025Decoupling Clinical and Class-Agnostic Features for Reliable Few-Shot Adaptation under Shiftcited by 3
  5. 2025Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Modelscited by 3
  6. 2025Class-Aware Prototype Learning with Negative Contrast for Test-Time Adaptation of Vision-Language Modelscited by 2
  7. 2025T3: Test-Time Model Merging in VLMs for Zero-Shot Medical Imaging Analysiscited by 2
  8. 2025Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Modelscited by 1
  9. 2026Can Experts Adapt Without Training? On Test-Time Modality Generalization in MVLMscited by 1
  10. 2024EFSA: Episodic Few-Shot Adaptation for Text-to-Image Retrievalcited by 0
  11. 2025Hierarchical Adaptive networks with Task vectors for Test-Time Adaptationcited by 0
  12. 2026Fair Context Learning for Evidence-Balanced Test-Time Adaptation in Vision-Language Modelscited by 0
5 more
  1. 2026Learning Inference Concurrency in DynamicGate MLP Structural and Mathematical Justificationcited by 0
  2. 2026Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation of SAM3 in Medical Image Segmentationcited by 0
  3. 2026MixTTA: Low-Rank Cross-Channel Mixing for Reliable Test-Time Adaptationcited by 0
  4. 2026Cross-device Collaborative Test-time Adaptation with Zeroth-order Optimization and Model Mergingcited by 0
  5. 2026Bridging the Confidence Gap: Temperature Scaling for Calibrating Test-Time Prompt Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Models
  2. 2025Realistic Test-Time Adaptation of Vision-Language Models
  3. 2025LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models
  4. 2024Low-Rank Few-Shot Adaptation of Vision-Language Models
  5. 2023Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models
  6. 2025The Illusion of Progress? A Critical Look at Test-Time Adaptation for Vision-Language Models
  7. 2025Efficient Test-Time Scaling for Small Vision-Language Models
  8. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?