# Open Set Domain Adaptation with Vision-language models via Gradient-aware Separation

arXiv: https://arxiv.org/abs/2505.13507 · HTML: https://mlsanity.com/paper/2505.13507/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 16 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Out-of-Distribution Detection with Deep Nearest Neighbors](https://mlsanity.com/paper/2204.06507/) (2022)
- [ViM: Out-Of-Distribution with Virtual-logit Matching](https://mlsanity.com/paper/2203.10807/) (2022)
- [On the Importance of Gradients for Detecting Distributional Shifts in the Wild](https://mlsanity.com/paper/2110.00218/) (2021)
- [Exploring the Limits of Out-of-Distribution Detection](https://mlsanity.com/paper/2106.03004/) (2021)
- [Delving into Out-of-Distribution Detection with Vision-Language Representations](https://mlsanity.com/paper/2211.13445/) (2022)
- [A Comprehensive Survey on Source-free Domain Adaptation](https://mlsanity.com/paper/2302.11803/) (2023)
- [VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model](https://mlsanity.com/paper/2501.12327/) (2025)
- [Domain-Agnostic Mutual Prompting for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2403.02899/) (2024)
- [GAIA: Delving into Gradient-based Attribution Abnormality for Out-of-distribution Detection](https://mlsanity.com/paper/2311.09620/) (2023)
- [Do we really have to filter out random noise in pre-training data for language models?](https://mlsanity.com/paper/2502.06604/) (2025)
- [Imbalanced Open Set Domain Adaptation via Moving-threshold Estimation and Gradual Alignment](https://mlsanity.com/paper/2303.04393/) (2023)
- [ATRI: Mitigating Multilingual Audio Text Retrieval Inconsistencies by Reducing Data Distribution Errors](https://mlsanity.com/paper/2502.14627/) (2025)

## Abstract (arXiv)

Open-Set Domain Adaptation (OSDA) confronts the dual challenge of aligning known-class distributions across domains while identifying target-domain-specific unknown categories. Current approaches often fail to leverage semantic relationships between modalities and struggle with error accumulation in unknown sample detection. We propose to harness Contrastive Language-Image Pretraining (CLIP) to address these limitations through two key innovations: 1) Prompt-driven cross-domain alignment: Learnable textual prompts conditioned on domain discrepancy metrics dynamically adapt CLIP's text encoder, enabling semantic consistency between source and target domains without explicit unknown-class supervision. 2) Gradient-aware open-set separation: A gradient analysis module quantifies domain shift by comparing the L2-norm of gradients from the learned prompts, where known/unknown samples exhibit statistically distinct gradient behaviors. Evaluations on Office-Home show that our method consistently outperforms CLIP baseline and standard baseline. Ablation studies confirm the gradient norm's critical role.