# ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free Domain Adaptation

arXiv: https://arxiv.org/abs/2308.03793 · HTML: https://mlsanity.com/paper/2308.03793/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 34 of its 61 references that are in the map.
- First posted on arXiv 4 Aug 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [Rotation Equivariant CNNs for Digital Pathology](https://mlsanity.com/paper/1806.03962/) (2018)
- [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1901.00976/) (2019)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [Semi-supervised Domain Adaptation via Minimax Entropy](https://mlsanity.com/paper/1904.06487/) (2019)
- [Label Propagation for Deep Semi-supervised Learning](https://mlsanity.com/paper/1904.04717/) (2019)
- [Domain Adaptation for Visual Applications: A Comprehensive Survey](https://mlsanity.com/paper/1702.05374/) (2017)
- [FixBi: Bridging Domain Spaces for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2011.09230/) (2020)
- [Unsupervised Domain Adaptation through Self-Supervision](https://mlsanity.com/paper/1909.11825/) (2019)

## Built on by (most cited first, 30 of 35)

- [Label Propagation for Zero-shot Classification with Vision-Language Models](https://mlsanity.com/paper/2404.04072/) (2024)
- [Mitigate the Gap: Investigating Approaches for Improving Cross-Modal Alignment in CLIP](https://mlsanity.com/paper/2406.17639/) (2024)
- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentation](https://mlsanity.com/paper/2503.19777/) (2025)
- [Semantic-Aligned Adversarial Evolution Triangle for High-Transferability Vision-Language Attack](https://mlsanity.com/paper/2411.02669/) (2024)
- [Source-Free Domain Adaptation Guided by Vision and Vision-Language Pre-Training](https://mlsanity.com/paper/2405.02954/) (2024)
- [CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey](https://mlsanity.com/paper/2504.14280/) (2025)
- [A Lost Opportunity for Vision-Language Models: A Comparative Study of Online Test-Time Adaptation for Vision-Language Models](https://mlsanity.com/paper/2405.14977/) (2024)
- [DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2404.14890/) (2024)
- [Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model](https://mlsanity.com/paper/2412.18303/) (2024)
- [Details Matter for Indoor Open-vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2507.23134/) (2025)
- [Generalizing vision-language models to novel domains: A comprehensive survey](https://mlsanity.com/paper/2506.18504/) (2025)
- [CoPT: Unsupervised Domain Adaptive Segmentation using Domain-Agnostic Text Embeddings](https://mlsanity.com/paper/2507.07125/) (2025)
- [Tuning-free Universally-Supervised Semantic Segmentation](https://mlsanity.com/paper/2405.14294/) (2024)
- [Simulations of Common Unsupervised Domain Adaptation Algorithms for Image Classification](https://mlsanity.com/paper/2502.10694/) (2025)
- [DPA: Dual Prototypes Alignment for Unsupervised Adaptation of Vision-Language Models](https://mlsanity.com/paper/2408.08855/) (2024)
- [GenZSL: Generative Zero-Shot Learning Via Inductive Variational Autoencoder](https://mlsanity.com/paper/2505.11882/) (2025)
- [Language-Guided Instance-Aware Domain-Adaptive Panoptic Segmentation](https://mlsanity.com/paper/2404.03799/) (2024)
- [Industrial Language-Image Dataset (ILID): Adapting Vision Foundation Models for Industrial Settings](https://mlsanity.com/paper/2406.09637/) (2024)
- [FLOSS: Free Lunch in Open-vocabulary Semantic Segmentation](https://mlsanity.com/paper/2504.10487/) (2025)
- [Empowering Source-Free Domain Adaptation via MLLM-Guided Reliability-Based Curriculum Learning](https://mlsanity.com/paper/2405.18376/) (2024)
- [LatteCLIP: Unsupervised CLIP Fine-Tuning via LMM-Synthetic Texts](https://mlsanity.com/paper/2410.08211/) (2024)
- [Federated Domain Generalization with Domain-specific Soft Prompts Generation](https://mlsanity.com/paper/2509.20807/) (2025)
- [BaFTA: Backprop-Free Test-Time Adaptation For Zero-Shot Vision-Language Models](https://mlsanity.com/paper/2406.11309/) (2024)
- [ViLAaD: Enhancing "Attracting and Dispersing'' Source-Free Domain Adaptation with Vision-and-Language Model](https://mlsanity.com/paper/2503.23529/) (2025)
- [LARE: Latent Augmentation using Regional Embedding with Vision-Language Model](https://mlsanity.com/paper/2409.12597/) (2024)
- [Towards Fine-Grained Adaptation of CLIP via a Self-Trained Alignment Score](https://mlsanity.com/paper/2507.09615/) (2025)
- [Noise-Tolerant Few-Shot Unsupervised Adapter for Vision-Language Models](https://mlsanity.com/paper/2309.14928/) (2023)
- [Exploiting Vision Language Model for Training-Free 3D Point Cloud OOD Detection via Graph Score Propagation](https://mlsanity.com/paper/2506.22375/) (2025)
- [Prototype-Guided Pseudo-Labeling with Neighborhood-Aware Consistency for Unsupervised Adaptation](https://mlsanity.com/paper/2507.22075/) (2025)

## Abstract (arXiv)

Large-scale Pre-Training Vision-Language Model such as CLIP has demonstrated outstanding performance in zero-shot classification, e.g. achieving 76.3% top-1 accuracy on ImageNet without seeing any example, which leads to potential benefits to many tasks that have no labeled data. However, while applying CLIP to a downstream target domain, the presence of visual and text domain gaps and cross-modality misalignment can greatly impact the model performance. To address such challenges, we propose ReCLIP, the first source-free domain adaptation method for vision-language models, which does not require any source data or target labeled data. ReCLIP first learns a projection space to mitigate the misaligned visual-text embeddings and learns pseudo labels, and then deploys cross-modality self-training with the pseudo labels, to update visual and text encoders, refine labels and reduce domain gaps and misalignments iteratively. With extensive experiments, we demonstrate ReCLIP reduces the average error rate of CLIP from 30.17% to 25.06% on 22 image classification benchmarks. Code available at https://github.com/michiganleon/ReCLIP_WACV.