# A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance

arXiv: https://arxiv.org/abs/2309.12530 · HTML: https://mlsanity.com/paper/2309.12530/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 68 of its 86 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 68)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [SWAD: Domain Generalization by Seeking Flat Minima](https://mlsanity.com/paper/2102.08604/) (2021)
- [Self-Challenging Improves Cross-Domain Generalization](https://mlsanity.com/paper/2007.02454/) (2020)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Unified Deep Supervised Domain Adaptation and Generalization](https://mlsanity.com/paper/1709.10190/) (2017)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Generalizing Across Domains via Cross-Gradient Training](https://mlsanity.com/paper/1804.10745/) (2018)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data](https://mlsanity.com/paper/1909.00889/) (2019)
- [Deep Domain-Adversarial Image Generation for Domain Generalisation](https://mlsanity.com/paper/2003.06054/) (2020)
- [Vision-Language Pre-Training with Triple Contrastive Learning](https://mlsanity.com/paper/2202.10401/) (2022)
- [Unified Contrastive Learning in Image-Text-Label Space](https://mlsanity.com/paper/2204.03610/) (2022)
- [DLOW: Domain Flow for Adaptation and Generalization](https://mlsanity.com/paper/1812.05418/) (2018)

## Built on by (most cited first, 30 of 40)

- [Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification](https://mlsanity.com/paper/2310.08255/) (2023)
- [Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language Models for Domain-Generalized Semantic Segmentation](https://mlsanity.com/paper/2504.03193/) (2025)
- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [Textual Query-Driven Mask Transformer for Domain Generalized Segmentation](https://mlsanity.com/paper/2407.09033/) (2024)
- [CLIP-Guided Generative Networks for Transferable Targeted Adversarial Attacks](https://mlsanity.com/paper/2407.10179/) (2024)
- [PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization](https://mlsanity.com/paper/2404.09011/) (2024)
- [CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey](https://mlsanity.com/paper/2504.14280/) (2025)
- [Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalization](https://mlsanity.com/paper/2407.15085/) (2024)
- [QT-DoG: Quantization-aware Training for Domain Generalization](https://mlsanity.com/paper/2410.06020/) (2024)
- [Diverse Target and Contribution Scheduling for Domain Generalization](https://mlsanity.com/paper/2309.16460/) (2023)
- [Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models](https://mlsanity.com/paper/2311.01441/) (2023)
- [Prompt-Driven Contrastive Learning for Transferable Adversarial Attacks](https://mlsanity.com/paper/2407.20657/) (2024)
- [When and How Does CLIP Enable Domain and Compositional Generalization?](https://mlsanity.com/paper/2502.09507/) (2025)
- [VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition](https://mlsanity.com/paper/2408.16930/) (2024)
- [Generalizing vision-language models to novel domains: A comprehensive survey](https://mlsanity.com/paper/2506.18504/) (2025)
- [CoPT: Unsupervised Domain Adaptive Segmentation using Domain-Agnostic Text Embeddings](https://mlsanity.com/paper/2507.07125/) (2025)
- [PromptTA: Prompt-driven Text Adapter for Source-free Domain Generalization](https://mlsanity.com/paper/2409.14163/) (2024)
- [OSLoPrompt: Bridging Low-Supervision Challenges and Open-Set Domain Generalization in CLIP](https://mlsanity.com/paper/2503.16106/) (2025)
- [Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions](https://mlsanity.com/paper/2506.11111/) (2025)
- [Tell, Don't Show!: Language Guidance Eases Transfer Across Domains in Images and Videos](https://mlsanity.com/paper/2403.05535/) (2024)
- [An Individual Identity-Driven Framework for Animal Re-Identification](https://mlsanity.com/paper/2410.22927/) (2024)
- [Domain Generalization via Discrete Codebook Learning](https://mlsanity.com/paper/2504.06572/) (2025)
- [Transitive Vision-Language Prompt Learning for Domain Generalization](https://mlsanity.com/paper/2404.18758/) (2024)
- [UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models](https://mlsanity.com/paper/2411.06921/) (2024)
- [Does CLIP perceive art the same way we do?](https://mlsanity.com/paper/2505.05229/) (2025)
- [Attention Head Purification: A New Perspective to Harness CLIP for Domain Generalization](https://mlsanity.com/paper/2412.07226/) (2024)
- [Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in CLIP](https://mlsanity.com/paper/2502.19269/) (2025)
- [LangDA: Building Context-Awareness via Language for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2503.12780/) (2025)
- [FEDTAIL: Federated Long-Tailed Domain Generalization with Sharpness-Guided Gradient Matching](https://mlsanity.com/paper/2506.08518/) (2025)
- [Generalizing Vision-Language Models with Dedicated Prompt Guidance](https://mlsanity.com/paper/2512.02421/) (2025)

## Abstract (arXiv)

Domain generalization studies the problem of training a model with samples from several domains (or distributions) and then testing the model with samples from a new, unseen domain. In this paper, we propose a novel approach for domain generalization that leverages recent advances in large vision-language models, specifically a CLIP teacher model, to train a smaller model that generalizes to unseen domains. The key technical contribution is a new type of regularization that requires the student's learned image representations to be close to the teacher's learned text representations obtained from encoding the corresponding text descriptions of images. We introduce two designs of the loss function, absolute and relative distance, which provide specific guidance on how the training process of the student model should be regularized. We evaluate our proposed method, dubbed RISE (Regularized Invariance with Semantic Embeddings), on various benchmark datasets and show that it outperforms several state-of-the-art domain generalization methods. To our knowledge, our work is the first to leverage knowledge distillation using a large vision-language model for domain generalization. By incorporating text-based information, RISE improves the generalization capability of machine learning models.