# Closing the Confusion Loop: CLIP-Guided Alignment for Source-Free Domain Adaptation

arXiv: https://arxiv.org/abs/2602.08730 · HTML: https://mlsanity.com/paper/2602.08730/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 42 of its 58 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [Exploiting Unintended Feature Leakage in Collaborative Learning](https://mlsanity.com/paper/1805.04049/) (2018)
- [VisDA: The Visual Domain Adaptation Challenge](https://mlsanity.com/paper/1710.06924/) (2017)
- [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1901.00976/) (2019)
- [Semi-supervised Domain Adaptation via Minimax Entropy](https://mlsanity.com/paper/1904.06487/) (2019)
- [Contrastive Test-Time Adaptation](https://mlsanity.com/paper/2204.10377/) (2022)
- [Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation](https://mlsanity.com/paper/2110.04202/) (2021)
- [Domain Adaptation via Prompt Learning](https://mlsanity.com/paper/2202.06687/) (2022)
- [Source-Free Domain Adaptation via Distribution Estimation](https://mlsanity.com/paper/2204.11257/) (2022)
- [Attracting and Dispersing: A Simple Approach for Source-free Domain Adaptation](https://mlsanity.com/paper/2205.04183/) (2022)
- [Guiding Pseudo-labels with Uncertainty Estimation for Source-free Unsupervised Domain Adaptation](https://mlsanity.com/paper/2303.03770/) (2023)
- [AD-CLIP: Adapting Domains in Prompt Space Using CLIP](https://mlsanity.com/paper/2308.05659/) (2023)
- [Confidence Score for Source-Free Unsupervised Domain Adaptation](https://mlsanity.com/paper/2206.06640/) (2022)
- [Source-Free Domain Adaptation with Frozen Multimodal Foundation Model](https://mlsanity.com/paper/2311.16510/) (2023)
- [C-SFDA: A Curriculum Learning Aided Self-Training Framework for Efficient Source Free Domain Adaptation](https://mlsanity.com/paper/2303.17132/) (2023)
- [When Source-Free Domain Adaptation Meets Learning with Noisy Labels](https://mlsanity.com/paper/2301.13381/) (2023)
- [Prompt-based Distribution Alignment for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2312.09553/) (2023)
- [BMD: A General Class-balanced Multicentric Dynamic Prototype Strategy for Source-free Domain Adaptation](https://mlsanity.com/paper/2204.02811/) (2022)
- [Invariant Feature Learning for Generalized Long-Tailed Classification](https://mlsanity.com/paper/2207.09504/) (2022)
- [Model Adaptation: Unsupervised Domain Adaptation without Source Data](https://mlsanity.com/paper/2502.19316/) (2025)

## Abstract (arXiv)

Source-Free Domain Adaptation (SFDA) tackles the problem of adapting a pre-trained source model to an unlabeled target domain without accessing any source data, which is quite suitable for the field of data security. Although recent advances have shown that pseudo-labeling strategies can be effective, they often fail in fine-grained scenarios due to subtle inter-class similarities. A critical but underexplored issue is the presence of asymmetric and dynamic class confusion, where visually similar classes are unequally and inconsistently misclassified by the source model. Existing methods typically ignore such confusion patterns, leading to noisy pseudo-labels and poor target discrimination. To address this, we propose CLIP-Guided Alignment(CGA), a novel framework that explicitly models and mitigates class confusion in SFDA. Generally, our method consists of three parts: (1) MCA: detects first directional confusion pairs by analyzing the predictions of the source model in the target domain; (2) MCC: leverages CLIP to construct confusion-aware textual prompts (e.g. a truck that looks like a bus), enabling more context-sensitive pseudo-labeling; and (3) FAM: builds confusion-guided feature banks for both CLIP and the source model and aligns them using contrastive learning to reduce ambiguity in the representation space. Extensive experiments on various datasets demonstrate that CGA consistently outperforms state-of-the-art SFDA methods, with especially notable gains in confusion-prone and fine-grained scenarios. Our results highlight the importance of explicitly modeling inter-class confusion for effective source-free adaptation. Our code can be find at https://github.com/soloiro/CGA