# Using Language to Extend to Unseen Domains

arXiv: https://arxiv.org/abs/2210.09520 · HTML: https://mlsanity.com/paper/2210.09520/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 42 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Simultaneous Deep Transfer Across Domains and Tasks](https://mlsanity.com/paper/1510.02192/) (2015)
- [Generate To Adapt: Aligning Domains using Generative Adversarial Networks](https://mlsanity.com/paper/1704.01705/) (2017)
- [Tell Me Where to Look: Guided Attention Inference Network](https://mlsanity.com/paper/1802.10171/) (2018)
- [Interpretations are useful: penalizing explanations to align neural networks with prior knowledge](https://mlsanity.com/paper/1909.13584/) (2019)
- [Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition](https://mlsanity.com/paper/1912.05534/) (2019)
- [Few-Shot Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.02536/) (2017)
- [Rewriting a Deep Generative Model](https://mlsanity.com/paper/2007.15646/) (2020)
- [Editing a classifier by rewriting its prediction rules](https://mlsanity.com/paper/2112.01008/) (2021)
- [On Guiding Visual Attention with Language Specification](https://mlsanity.com/paper/2202.08926/) (2022)

## Built on by (most cited first, 30 of 34)

- [Fake it till you make it: Learning transferable representations from synthetic ImageNet clones](https://mlsanity.com/paper/2212.08420/) (2022)
- [Waffling around for Performance: Visual Classification with Random Words and Broad Concepts](https://mlsanity.com/paper/2306.07282/) (2023)
- [Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation](https://mlsanity.com/paper/2305.16289/) (2023)
- [PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization](https://mlsanity.com/paper/2307.15199/) (2023)
- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [LanDA: Language-Guided Multi-Source Domain Adaptation](https://mlsanity.com/paper/2401.14148/) (2024)
- [Interactive Data Synthesis for Systematic Vision Adaptation via LLMs-AIGCs Collaboration](https://mlsanity.com/paper/2305.12799/) (2023)
- [Where Does My Model Underperform? A Human Evaluation of Slice Discovery Algorithms](https://mlsanity.com/paper/2306.08167/) (2023)
- [Prompt-Driven Contrastive Learning for Transferable Adversarial Attacks](https://mlsanity.com/paper/2407.20657/) (2024)
- [GeNIe: Generative Hard Negative Images Through Diffusion](https://mlsanity.com/paper/2312.02548/) (2023)
- [Clarify: Improving Model Robustness With Natural Language Corrections](https://mlsanity.com/paper/2402.03715/) (2024)
- [DPStyler: Dynamic PromptStyler for Source-Free Domain Generalization](https://mlsanity.com/paper/2403.16697/) (2024)
- [ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations](https://mlsanity.com/paper/2308.10103/) (2023)
- [Lightweight Unsupervised Federated Learning with Pretrained Vision Language Model](https://mlsanity.com/paper/2404.11046/) (2024)
- [Coping with Change: Learning Invariant and Minimum Sufficient Representations for Fine-Grained Visual Categorization](https://mlsanity.com/paper/2306.04893/) (2023)
- [Tell, Don't Show!: Language Guidance Eases Transfer Across Domains in Images and Videos](https://mlsanity.com/paper/2403.05535/) (2024)
- [Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation](https://mlsanity.com/paper/2409.08598/) (2024)
- [UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models](https://mlsanity.com/paper/2411.06921/) (2024)
- [MaskMedPaint: Masked Medical Image Inpainting with Diffusion Models for Mitigation of Spurious Correlations](https://mlsanity.com/paper/2411.10686/) (2024)
- [Explaining Domain Shifts in Language: Concept erasing for Interpretable Image Classification](https://mlsanity.com/paper/2503.18483/) (2025)
- [Language-assisted Vision Model Debugger: A Sample-Free Approach to Finding and Fixing Bugs](https://mlsanity.com/paper/2312.05588/) (2023)
- [Cross-modality debiasing: using language to mitigate sub-population shifts in imaging](https://mlsanity.com/paper/2403.07888/) (2024)
- [WIDIn: Wording Image for Domain-Invariant Representation in Single-Source Domain Generalization](https://mlsanity.com/paper/2405.18405/) (2024)
- [LARE: Latent Augmentation using Regional Embedding with Vision-Language Model](https://mlsanity.com/paper/2409.12597/) (2024)
- [An Empirical Study of Validating Synthetic Data for Text-Based Person Retrieval](https://mlsanity.com/paper/2503.22171/) (2025)
- [LaViP:Language-Grounded Visual Prompts](https://mlsanity.com/paper/2312.10945/) (2023)
- [SAFLEX: Self-Adaptive Augmentation via Feature Label Extrapolation](https://mlsanity.com/paper/2410.02512/) (2024)
- [Semantically Guided Action Anticipation](https://mlsanity.com/paper/2411.15557/) (2024)
- [Masked Language Prompting for Generative Data Augmentation in Few-shot Fashion Style Recognition](https://mlsanity.com/paper/2504.19455/) (2025)
- [TRUST: Leveraging Text Robustness for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2508.06452/) (2025)

## Abstract (arXiv)

It is expensive to collect training data for every possible domain that a vision model may encounter when deployed. We instead consider how simply verbalizing the training domain (e.g. "photos of birds") as well as domains we want to extend to but do not have data for (e.g. "paintings of birds") can improve robustness. Using a multimodal model with a joint image and language embedding space, our method LADS learns a transformation of the image embeddings from the training domain to each unseen test domain, while preserving task relevant information. Without using any images from the unseen test domain, we show that over the extended domain containing both training and unseen test domains, LADS outperforms standard fine-tuning and ensemble approaches over a suite of four benchmarks targeting domain adaptation and dataset bias.