# Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimization

arXiv: https://arxiv.org/abs/2308.12919 · HTML: https://mlsanity.com/paper/2308.12919/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 52 references that are in the map.
- First posted on arXiv 24 Aug 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 13 of 13)

- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [VisDA: The Visual Domain Adaptation Challenge](https://mlsanity.com/paper/1710.06924/) (2017)
- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.15647/) (2023)
- [Unified Vision and Language Prompt Learning](https://mlsanity.com/paper/2210.07225/) (2022)
- [Unsupervised Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2204.03649/) (2022)
- [LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning](https://mlsanity.com/paper/2306.01293/) (2023)
- [How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?](https://mlsanity.com/paper/2306.06048/) (2023)
- [UMAD: Universal Model Adaptation under Domain and Category Shift](https://mlsanity.com/paper/2112.08553/) (2021)
- [AUTO: Adaptive Outlier Optimization for Test-Time OOD Detection](https://mlsanity.com/paper/2303.12267/) (2023)
- [M-Tuning: Prompt Tuning with Mitigated Label Bias in Open-Set Scenarios](https://mlsanity.com/paper/2303.05122/) (2023)

## Built on by (most cited first, 13 of 13)

- [LoRA-Pro: Are Low-Rank Adapters Properly Optimized?](https://mlsanity.com/paper/2407.18242/) (2024)
- [Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey](https://mlsanity.com/paper/2407.21794/) (2024)
- [COME: Test-time adaption by Conservatively Minimizing Entropy](https://mlsanity.com/paper/2410.10894/) (2024)
- [Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advances](https://mlsanity.com/paper/2409.11884/) (2024)
- [CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey](https://mlsanity.com/paper/2504.14280/) (2025)
- [STAMP: Outlier-Aware Test-Time Adaptation with Stable Memory Replay](https://mlsanity.com/paper/2407.15773/) (2024)
- [Generalizing vision-language models to novel domains: A comprehensive survey](https://mlsanity.com/paper/2506.18504/) (2025)
- [QueryAdapter: Rapid Adaptation of Vision-Language Models in Response to Natural Language Queries](https://mlsanity.com/paper/2502.18735/) (2025)
- [Generalizing Vision-Language Models with Dedicated Prompt Guidance](https://mlsanity.com/paper/2512.02421/) (2025)
- [Beyond Retraining: Training-Free Unknown Class Filtering for Source-Free Open Set Domain Adaptation of Vision-Language Models](https://mlsanity.com/paper/2504.14224/) (2025)
- [Frustratingly Easy Feature Reconstruction for Out-of-Distribution Detection](https://mlsanity.com/paper/2509.06988/) (2025)
- [Cooperative Pseudo Labeling for Unsupervised Federated Classification](https://mlsanity.com/paper/2510.10100/) (2025)
- [Bridging the Confidence Gap: Temperature Scaling for Calibrating Test-Time Prompt Tuning](https://mlsanity.com/paper/2609.17386/) (2026)

## Abstract (arXiv)

The emergence of vision-language models, such as CLIP, has spurred a significant research effort towards their application for downstream supervised learning tasks. Although some previous studies have explored the unsupervised fine-tuning of CLIP, they often rely on prior knowledge in the form of class names associated with ground truth labels. This paper explores a realistic unsupervised fine-tuning scenario, considering the presence of out-of-distribution samples from unknown classes within the unlabeled data. In particular, we focus on simultaneously enhancing out-of-distribution detection and the recognition of instances associated with known classes. To tackle this problem, we present a simple, efficient, and effective approach called Universal Entropy Optimization (UEO). UEO leverages sample-level confidence to approximately minimize the conditional entropy of confident instances and maximize the marginal entropy of less confident instances. Apart from optimizing the textual prompt, UEO incorporates optimization of channel-wise affine transformations within the visual branch of CLIP. Extensive experiments across 15 domains and 4 different types of prior knowledge validate the effectiveness of UEO compared to baseline methods. The code is publicly available at urlhttps://github.com/tim-learn/UEO.