# Domain Generalization by Mutual-Information Regularization with Pre-trained Models

arXiv: https://arxiv.org/abs/2203.10789 · HTML: https://mlsanity.com/paper/2203.10789/

## Key facts

- Cited by 133 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 76 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 30 of 63)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [MINE: Mutual Information Neural Estimation](https://mlsanity.com/paper/1801.04062/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [Domain Generalization with MixStyle](https://mlsanity.com/paper/2104.02008/) (2021)
- [Domain Generalization by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1903.06864/) (2019)
- [SWAD: Domain Generalization by Seeking Flat Minima](https://mlsanity.com/paper/2102.08604/) (2021)
- [Self-Challenging Improves Cross-Domain Generalization](https://mlsanity.com/paper/2007.02454/) (2020)

## Built on by (most cited first, 30 of 133)

- [Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs](https://mlsanity.com/paper/2212.00785/) (2022)
- [Sharpness-Aware Gradient Matching for Domain Generalization](https://mlsanity.com/paper/2303.10353/) (2023)
- [CLIPood: Generalizing CLIP to Out-of-Distributions](https://mlsanity.com/paper/2302.00864/) (2023)
- [Sparse Mixture-of-Experts are Domain Generalizable Learners](https://mlsanity.com/paper/2206.04046/) (2022)
- [Nearest Neighbor Guidance for Out-of-Distribution Detection](https://mlsanity.com/paper/2309.14888/) (2023)
- [PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization](https://mlsanity.com/paper/2307.15199/) (2023)
- [Read-only Prompt Optimization for Vision-Language Few-shot Learning](https://mlsanity.com/paper/2308.14960/) (2023)
- [Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification](https://mlsanity.com/paper/2310.08255/) (2023)
- [A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance](https://mlsanity.com/paper/2309.12530/) (2023)
- [StyLIP: Multi-Scale Style-Conditioned Prompt Learning for CLIP-based Domain Generalization](https://mlsanity.com/paper/2302.09251/) (2023)
- [DiPrompT: Disentangled Prompt Tuning for Multiple Latent Domain Generalization in Federated Learning](https://mlsanity.com/paper/2403.08506/) (2024)
- [Exploring Visual Interpretability for Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2209.07046/) (2022)
- [Context-Aware Robust Fine-Tuning](https://mlsanity.com/paper/2211.16175/) (2022)
- [Flatness-Aware Minimization for Domain Generalization](https://mlsanity.com/paper/2307.11108/) (2023)
- [Generalizable Decision Boundaries: Dualistic Meta-Learning for Open Set Domain Generalization](https://mlsanity.com/paper/2308.09391/) (2023)
- [Soft Prompt Generation for Domain Generalization](https://mlsanity.com/paper/2404.19286/) (2024)
- [Rethinking Multi-domain Generalization with A General Learning Objective](https://mlsanity.com/paper/2402.18853/) (2024)
- [Adapting to Distribution Shift by Visual Domain Prompt Generation](https://mlsanity.com/paper/2405.02797/) (2024)
- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [MADG: Margin-based Adversarial Learning for Domain Generalization](https://mlsanity.com/paper/2311.08503/) (2023)
- [Domain-Unified Prompt Representations for Source-Free Domain Generalization](https://mlsanity.com/paper/2209.14926/) (2022)
- [DomainDrop: Suppressing Domain-Sensitive Channels for Domain Generalization](https://mlsanity.com/paper/2308.10285/) (2023)
- [Proxy Denoising for Source-Free Domain Adaptation](https://mlsanity.com/paper/2406.01658/) (2024)
- [DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networks](https://mlsanity.com/paper/2302.14685/) (2023)
- [Rethinking the Evaluation Protocol of Domain Generalization](https://mlsanity.com/paper/2305.15253/) (2023)
- [MI-SegNet: Mutual Information-Based US Segmentation for Unseen Domain Generalization](https://mlsanity.com/paper/2303.12649/) (2023)
- [ERM++: An Improved Baseline for Domain Generalization](https://mlsanity.com/paper/2304.01973/) (2023)
- [CHAMMI: A benchmark for channel-adaptive models in microscopy imaging](https://mlsanity.com/paper/2310.19224/) (2023)
- [Domain Generalization via Rationale Invariance](https://mlsanity.com/paper/2308.11158/) (2023)
- [PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization](https://mlsanity.com/paper/2404.09011/) (2024)

## Abstract (arXiv)

Domain generalization (DG) aims to learn a generalized model to an unseen target domain using only limited source domains. Previous attempts to DG fail to learn domain-invariant representations only from the source domains due to the significant domain shifts between training and test domains. Instead, we re-formulate the DG objective using mutual information with the oracle model, a model generalized to any possible domain. We derive a tractable variational lower bound via approximating the oracle model by a pre-trained model, called Mutual Information Regularization with Oracle (MIRO). Our extensive experiments show that MIRO significantly improves the out-of-distribution performance. Furthermore, our scaling experiments show that the larger the scale of the pre-trained model, the greater the performance improvement of MIRO. Source code is available at https://github.com/kakaobrain/miro.