# USE: A Unified Self-Ensembling Framework for Test-Time Prompt Tuning

arXiv: https://arxiv.org/abs/2607.03900 · HTML: https://mlsanity.com/paper/2607.03900/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 5 of its 76 references that are in the map.
- First posted on arXiv 4 Jul 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 5 of 5)

- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Unsupervised Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2204.03649/) (2022)
- [What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time](https://mlsanity.com/paper/2603.19880/) (2026)
- [Adapting Vision-Language Models Without Labels: A Comprehensive Survey](https://mlsanity.com/paper/2508.05547/) (2025)
- [Balanced Symmetric Cross Entropy for Large Scale Imbalanced and Noisy Data](https://mlsanity.com/paper/2007.01618/) (2020)

## Abstract (arXiv)

Test-time adaptation (TTA) has emerged as a popular paradigm for improving the performance of vision-language models (e.g., CLIP) on downstream tasks. Among existing CLIP-based TTA methods, Test-Time Prompt Tuning (TPT) is a pioneering work that optimizes textual prompts using multiple test-time augmentations and remains a strong baseline to date. In this work, we revisit TPT and reveal that its optimization can be interpreted as implicitly learning from self-generated pseudo labels. Building on this perspective, we propose a unified self-ensembling framework (USE) that ensures consistency between the optimization and inference stages. During optimization, we introduce a simple yet effective self-ensembling (SE) strategy that emphasizes the test image itself over its augmented views adaptively to obtain more reliable pseudo labels. To fully exploit the potential of augmentations, we further apply the same strategy at inference time, unifying the objectives of both stages. Notably, SE can also act as a lightweight optimization-free TTA method. Extensive experiments across multiple datasets demonstrate that SE and USE outperform their counterparts, respectively. Furthermore, SE yields consistent performance gains when integrated with existing TTA methods. The code is available at https://github.com/sirujiang/USE.