# Preserving Clusters in Prompt Learning for Unsupervised Domain Adaptation

arXiv: https://arxiv.org/abs/2506.11493 · HTML: https://mlsanity.com/paper/2506.11493/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 65 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1903.04064/) (2019)
- [A DIRT-T Approach to Unsupervised Domain Adaptation](https://mlsanity.com/paper/1802.08735/) (2018)
- [Simultaneous Deep Transfer Across Domains and Tasks](https://mlsanity.com/paper/1510.02192/) (2015)
- [Unsupervised Domain Adaptation via Structurally Regularized Deep Clustering](https://mlsanity.com/paper/2003.08607/) (2020)
- [Towards Discriminability and Diversity: Batch Nuclear-norm Maximization under Label Insufficient Situations](https://mlsanity.com/paper/2003.12237/) (2020)
- [Gradually Vanishing Bridge for Adversarial Domain Adaptation](https://mlsanity.com/paper/2003.13183/) (2020)
- [Domain Adaptation via Prompt Learning](https://mlsanity.com/paper/2202.06687/) (2022)
- [Domain-Symmetric Networks for Adversarial Domain Adaptation](https://mlsanity.com/paper/1904.04663/) (2019)
- [Deep Cocktail Network: Multi-source Unsupervised Domain Adaptation with Category Shift](https://mlsanity.com/paper/1803.00830/) (2018)
- [Multi-source Distilling Domain Adaptation](https://mlsanity.com/paper/1911.11554/) (2019)
- [Learning Disentangled Semantic Representation for Domain Adaptation](https://mlsanity.com/paper/2012.11807/) (2020)
- [Unsupervised Domain Adaptation via Structured Prediction Based Selective Pseudo-Labeling](https://mlsanity.com/paper/1911.07982/) (2019)
- [Learning to Combine: Knowledge Aggregation for Multi-Source Domain Adaptation](https://mlsanity.com/paper/2007.08801/) (2020)
- [Your Classifier can Secretly Suffice Multi-Source Domain Adaptation](https://mlsanity.com/paper/2103.11169/) (2021)

## Built on by (most cited first, 4 of 4)

- [Fed-ADE: Adaptive Learning Rate for Federated Post-adaptation under Distribution Shift](https://mlsanity.com/paper/2603.01040/) (2026)
- [Dance Across Shifts: Forward-Facilitation Continual Test-Time Adaptation through Dynamic Style Bridging](https://mlsanity.com/paper/2605.18608/) (2026)
- [Masked Representation Modeling for Domain-Adaptive Segmentation](https://mlsanity.com/paper/2509.13801/) (2025)
- [Geometric-Aware Hypergraph Reasoning for Novel Class Discovery in Point Cloud Segmentation](https://mlsanity.com/paper/2606.07280/) (2026)

## Abstract (arXiv)

Recent approaches leveraging multi-modal pre-trained models like CLIP for Unsupervised Domain Adaptation (UDA) have shown significant promise in bridging domain gaps and improving generalization by utilizing rich semantic knowledge and robust visual representations learned through extensive pre-training on diverse image-text datasets. While these methods achieve state-of-the-art performance across benchmarks, much of the improvement stems from base pseudo-labels (CLIP zero-shot predictions) and self-training mechanisms. Thus, the training mechanism exhibits a key limitation wherein the visual embedding distribution in target domains can deviate from the visual embedding distribution in the pre-trained model, leading to misguided signals from class descriptions. This work introduces a fresh solution to reinforce these pseudo-labels and facilitate target-prompt learning, by exploiting the geometry of visual and text embeddings - an aspect that is overlooked by existing methods. We first propose to directly leverage the reference predictions (from source prompts) based on the relationship between source and target visual embeddings. We later show that there is a strong clustering behavior observed between visual and text embeddings in pre-trained multi-modal models. Building on optimal transport theory, we transform this insight into a novel strategy to enforce the clustering property in text embeddings, further enhancing the alignment in the target domain. Our experiments and ablation studies validate the effectiveness of the proposed approach, demonstrating superior performance and improved quality of target prompts in terms of representation.