# Domain-Controlled Prompt Learning

arXiv: https://arxiv.org/abs/2310.07730 · HTML: https://mlsanity.com/paper/2310.07730/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 57 references that are in the map.
- First posted on arXiv 30 Sep 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [Zero-Shot Learning Through Cross-Modal Transfer](https://mlsanity.com/paper/1301.3666/) (2013)
- [ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias](https://mlsanity.com/paper/2106.03348/) (2021)
- [Exploring Models and Data for Remote Sensing Image Caption Generation](https://mlsanity.com/paper/1712.07835/) (2017)

## Built on by (most cited first, 22 of 22)

- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees](https://mlsanity.com/paper/2406.07115/) (2024)
- [BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models](https://mlsanity.com/paper/2411.15232/) (2024)
- [Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation](https://mlsanity.com/paper/2403.11808/) (2024)
- [Domain Prompt Learning with Quaternion Networks](https://mlsanity.com/paper/2312.08878/) (2023)
- [One is Plenty: A Polymorphic Feature Interpreter for Immutable Heterogeneous Collaborative Perception](https://mlsanity.com/paper/2411.16799/) (2024)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [CoPT: Unsupervised Domain Adaptive Segmentation using Domain-Agnostic Text Embeddings](https://mlsanity.com/paper/2507.07125/) (2025)
- [TextSAM-EUS: Text Prompt Learning for SAM to Accurately Segment Pancreatic Tumor in Endoscopic Ultrasound](https://mlsanity.com/paper/2507.18082/) (2025)
- [Medical Knowledge Intervention Prompt Tuning for Medical Image Classification](https://mlsanity.com/paper/2511.12639/) (2025)
- [UPRE: Zero-Shot Domain Adaptation for Object Detection via Unified Prompt and Representation Enhancement](https://mlsanity.com/paper/2507.00721/) (2025)
- [MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation](https://mlsanity.com/paper/2602.20423/) (2026)
- [Foundation Models for Anomaly Detection: Vision and Challenges](https://mlsanity.com/paper/2502.06911/) (2025)
- [MGPATH: Vision-Language Model with Multi-Granular Prompt Learning for Few-Shot WSI Classification](https://mlsanity.com/paper/2502.07409/) (2025)
- [Generalizable Prompt Learning of CLIP: A Brief Overview](https://mlsanity.com/paper/2503.01263/) (2025)
- [CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Values](https://mlsanity.com/paper/2509.03740/) (2025)
- [Advancing Prompt Learning through an External Layer](https://mlsanity.com/paper/2407.19674/) (2024)
- [FrogDogNet: Fourier frequency Retained visual prompt Output Guidance for Domain Generalization of CLIP in Remote Sensing](https://mlsanity.com/paper/2504.16433/) (2025)
- [SciTopic: Enhancing Topic Discovery in Scientific Literature through Advanced LLM](https://mlsanity.com/paper/2508.20514/) (2025)
- [Brain Imaging Foundation Models, Are We There Yet? A Systematic Review of Foundation Models for Brain Imaging and Biomedical Research](https://mlsanity.com/paper/2506.13306/) (2025)
- [Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification](https://mlsanity.com/paper/2604.23977/) (2026)
- [PEARL: A Lightweight Prompt-based Feature Interpreter Framework for Real-Time, Anonymous, and Heterogeneous Collaborative Perception](https://mlsanity.com/paper/2609.27123/) (2026)

## Abstract (arXiv)

Large pre-trained vision-language models, such as CLIP, have shown remarkable generalization capabilities across various tasks when appropriate text prompts are provided. However, adapting these models to specific domains, like remote sensing images (RSIs), medical images, etc, remains unexplored and challenging. Existing prompt learning methods often lack domain-awareness or domain-transfer mechanisms, leading to suboptimal performance due to the misinterpretation of specific images in natural image patterns. To tackle this dilemma, we proposed a Domain-Controlled Prompt Learning for the specific domains. Specifically, the large-scale specific domain foundation model (LSDM) is first introduced to provide essential specific domain knowledge. Using lightweight neural networks, we transfer this knowledge into domain biases, which control both the visual and language branches to obtain domain-adaptive prompts in a directly incorporating manner. Simultaneously, to overcome the existing overfitting challenge, we propose a novel noisy-adding strategy, without extra trainable parameters, to help the model escape the suboptimal solution in a global domain oscillation manner. Experimental results show our method achieves state-of-the-art performance in specific domain image recognition datasets. Our code is available at https://github.com/caoql98/DCPL.