# Prompt-based Distribution Alignment for Unsupervised Domain Adaptation

arXiv: https://arxiv.org/abs/2312.09553 · HTML: https://mlsanity.com/paper/2312.09553/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 15 Dec 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [A Survey of Unsupervised Deep Domain Adaptation](https://mlsanity.com/paper/1812.02849/) (2018)
- [Bridging Theory and Algorithm for Domain Adaptation](https://mlsanity.com/paper/1904.05801/) (2019)
- [Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning](https://mlsanity.com/paper/2104.03135/) (2021)
- [Unsupervised Domain Adaptation via Structurally Regularized Deep Clustering](https://mlsanity.com/paper/2003.08607/) (2020)
- [CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention](https://mlsanity.com/paper/2209.14169/) (2022)
- [A Closer Look at Smoothness in Domain Adversarial Training](https://mlsanity.com/paper/2206.08213/) (2022)
- [Safe Self-Refinement for Transformer-based Domain Adaptation](https://mlsanity.com/paper/2204.07683/) (2022)
- [Pareto Self-Supervised Training for Few-Shot Learning](https://mlsanity.com/paper/2104.07841/) (2021)
- [Learning How to Propagate Messages in Graph Neural Networks](https://mlsanity.com/paper/2310.00697/) (2023)
- [Tree Structure-Aware Few-Shot Image Classification via Hierarchical Aggregation](https://mlsanity.com/paper/2207.06989/) (2022)
- [Generalized Universal Domain Adaptation with Generative Flow Networks](https://mlsanity.com/paper/2305.04466/) (2023)
- [RotoGBML: Towards Out-of-Distribution Generalization for Gradient-Based Meta-Learning](https://mlsanity.com/paper/2303.06679/) (2023)

## Built on by (most cited first, 30 of 44)

- [Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective](https://mlsanity.com/paper/2404.19287/) (2024)
- [Soft Prompt Generation for Domain Generalization](https://mlsanity.com/paper/2404.19286/) (2024)
- [Proxy Denoising for Source-Free Domain Adaptation](https://mlsanity.com/paper/2406.01658/) (2024)
- [CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey](https://mlsanity.com/paper/2504.14280/) (2025)
- [Anchor-based Robust Finetuning of Vision-Language Models](https://mlsanity.com/paper/2404.06244/) (2024)
- [M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension](https://mlsanity.com/paper/2407.01131/) (2024)
- [Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Models](https://mlsanity.com/paper/2406.10584/) (2024)
- [Generalizing vision-language models to novel domains: A comprehensive survey](https://mlsanity.com/paper/2506.18504/) (2025)
- [Debiased Collaborative Filtering with Kernel-Based Causal Balancing](https://mlsanity.com/paper/2404.19596/) (2024)
- [AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation](https://mlsanity.com/paper/2403.07030/) (2024)
- [MetaCoCo: A New Few-Shot Classification Benchmark with Spurious Correlation](https://mlsanity.com/paper/2404.19644/) (2024)
- [Reclaiming Lost Text Layers for Source-Free Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2603.05235/) (2026)
- [Unified modality separation: A vision-language framework for unsupervised domain adaptation](https://mlsanity.com/paper/2508.04987/) (2025)
- [Mind the Discriminability Trap in Source-Free Cross-domain Few-shot Learning](https://mlsanity.com/paper/2603.13341/) (2026)
- [Deep Generative Domain Adaptation with Temporal Attention for Cross-User Activity Recognition](https://mlsanity.com/paper/2403.17958/) (2024)
- [An Individual Identity-Driven Framework for Animal Re-Identification](https://mlsanity.com/paper/2410.22927/) (2024)
- [Generalizable Multispectral Land Cover Classification via Frequency-Aware Mixture of Low-Rank Token Experts](https://mlsanity.com/paper/2505.14088/) (2025)
- [Be Aware of the Neighborhood Effect: Modeling Selection Bias under Interference](https://mlsanity.com/paper/2404.19620/) (2024)
- [Data-Efficient CLIP-Powered Dual-Branch Networks for Source-Free Unsupervised Domain Adaptation](https://mlsanity.com/paper/2410.15811/) (2024)
- [Adaptive Prompt Learning with Negative Textual Semantics and Uncertainty Modeling for Universal Multi-Source Domain Adaptation](https://mlsanity.com/paper/2404.14696/) (2024)
- [VirDA: Reusing Backbone for Unsupervised Domain Adaptation with Visual Reprogramming](https://mlsanity.com/paper/2510.01660/) (2025)
- [Mitigating Negative Transfer via Reducing Environmental Disagreement](https://mlsanity.com/paper/2510.24044/) (2025)
- [Collaborative Learning with Multiple Foundation Models for Source-Free Domain Adaptation](https://mlsanity.com/paper/2511.19147/) (2025)
- [Dual-Path Stable Soft Prompt Generation for Domain Generalization](https://mlsanity.com/paper/2505.18770/) (2025)
- [Collaborative Editable Model](https://mlsanity.com/paper/2506.14146/) (2025)
- [Multi-Prompt Progressive Alignment for Multi-Source Unsupervised Domain Adaptation](https://mlsanity.com/paper/2507.23373/) (2025)
- [Set Pivot Learning: Redefining Generalized Segmentation with Vision Foundation Models](https://mlsanity.com/paper/2508.01582/) (2025)
- [Leveraging Unlabeled Data from Unknown Sources via Dual-Path Guidance for Deepfake Face Detection](https://mlsanity.com/paper/2508.09022/) (2025)
- [Feature-Space Planes Searcher: A Universal Domain Adaptation Framework for Interpretability and Computational Efficiency](https://mlsanity.com/paper/2508.18693/) (2025)
- [SynthGenNet: a self-supervised approach for test-time generalization using synthetic multi-source domain mixing of street view images](https://mlsanity.com/paper/2509.02287/) (2025)

## Abstract (arXiv)

Recently, despite the unprecedented success of large pre-trained visual-language models (VLMs) on a wide range of downstream tasks, the real-world unsupervised domain adaptation (UDA) problem is still not well explored. Therefore, in this paper, we first experimentally demonstrate that the unsupervised-trained VLMs can significantly reduce the distribution discrepancy between source and target domains, thereby improving the performance of UDA. However, a major challenge for directly deploying such models on downstream UDA tasks is prompt engineering, which requires aligning the domain knowledge of source and target domains, since the performance of UDA is severely influenced by a good domain-invariant representation. We further propose a Prompt-based Distribution Alignment (PDA) method to incorporate the domain knowledge into prompt learning. Specifically, PDA employs a two-branch prompt-tuning paradigm, namely base branch and alignment branch. The base branch focuses on integrating class-related representation into prompts, ensuring discrimination among different classes. To further minimize domain discrepancy, for the alignment branch, we construct feature banks for both the source and target domains and propose image-guided feature tuning (IFT) to make the input attend to feature banks, which effectively integrates self-enhanced and cross-domain features into the model. In this way, these two branches can be mutually promoted to enhance the adaptation of VLMs for UDA. We conduct extensive experiments on three benchmarks to demonstrate that our proposed PDA achieves state-of-the-art performance. The code is available at https://github.com/BaiShuanghao/Prompt-based-Distribution-Alignment.