# Noise is an Efficient Learner for Zero-Shot Vision-Language Models

arXiv: https://arxiv.org/abs/2502.06019 · HTML: https://mlsanity.com/paper/2502.06019/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 9 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Continual Test-Time Domain Adaptation](https://mlsanity.com/paper/2203.13591/) (2022)
- [Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2209.07511/) (2022)
- [MEMO: Test Time Robustness via Adaptation and Augmentation](https://mlsanity.com/paper/2110.09506/) (2021)
- [Efficient Test-Time Adaptation of Vision-Language Models](https://mlsanity.com/paper/2403.18293/) (2024)
- [CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention](https://mlsanity.com/paper/2209.14169/) (2022)
- [C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion](https://mlsanity.com/paper/2403.14119/) (2024)
- [BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP](https://mlsanity.com/paper/2311.16194/) (2023)
- [On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?](https://mlsanity.com/paper/2405.02266/) (2024)
- [Beyond Model Adaptation at Test Time: A Survey](https://mlsanity.com/paper/2411.03687/) (2024)
- [Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2305.18010/) (2023)
- [Robust Calibration of Large Vision-Language Adapters](https://mlsanity.com/paper/2407.13588/) (2024)
- [An Empirical Study Into What Matters for Calibrating Vision-Language Models](https://mlsanity.com/paper/2402.07417/) (2024)
- [Test-Time Low Rank Adaptation via Confidence Maximization for Zero-Shot Generalization of Vision-Language Models](https://mlsanity.com/paper/2407.15913/) (2024)
- [BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning](https://mlsanity.com/paper/2408.07440/) (2024)

## Built on by (most cited first, 6 of 6)

- [On the Robustness of Medical Vision-Language Models: Are they Truly Generalizable?](https://mlsanity.com/paper/2505.15425/) (2025)
- [Adapting Vision-Language Models Without Labels: A Comprehensive Survey](https://mlsanity.com/paper/2508.05547/) (2025)
- [Decoupling Clinical and Class-Agnostic Features for Reliable Few-Shot Adaptation under Shift](https://mlsanity.com/paper/2509.09397/) (2025)
- [Can Experts Adapt Without Training? On Test-Time Modality Generalization in MVLMs](https://mlsanity.com/paper/2607.16726/) (2026)
- [ZEBRA: Zero-Shot Entropy-Regularized Prompt Learning for Base-to-Novel Generalization in Audio-Language Models](https://mlsanity.com/paper/2606.31587/) (2026)
- [Harnessing SPAD Noise: Test-Time Gated Attention for Indoor Depth Estimation](https://mlsanity.com/paper/2610.15431/) (2026)

## Abstract (arXiv)

Recently, test-time adaptation has garnered attention as a method for tuning models without labeled data. The conventional modus operandi for adapting pre-trained vision-language models (VLMs) during test-time primarily focuses on tuning learnable prompts; however, this approach overlooks potential distribution shifts in the visual representations themselves. In this work, we address this limitation by introducing Test-Time Noise Tuning (TNT), a novel method for handling unpredictable shifts in the visual space. TNT leverages, for the first time, a noise adaptation strategy that optimizes learnable noise directly in the visual input space, enabling adaptive feature learning from a single test sample. We further introduce a novel approach for inter-view representation alignment by explicitly enforcing coherence in embedding distances, ensuring consistent feature representations across views. Combined with scaled logits and confident view selection at inference, TNT substantially enhances VLM generalization and calibration, achieving average gains of +7.38% on natural distributions benchmark and +0.80% on cross-dataset evaluations over zero-shot CLIP. These improvements lay a strong foundation for adaptive out-of-distribution handling.