# Domain Generalization using Pretrained Models without Fine-tuning

arXiv: https://arxiv.org/abs/2203.04600 · HTML: https://mlsanity.com/paper/2203.04600/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 58 references that are in the map.
- First posted on arXiv 9 Mar 2022.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [Domain Generalization by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1903.06864/) (2019)
- [SWAD: Domain Generalization by Seeking Flat Minima](https://mlsanity.com/paper/2102.08604/) (2021)
- [Domain Generalization via Model-Agnostic Learning of Semantic Features](https://mlsanity.com/paper/1910.13580/) (2019)
- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Gradient Matching for Domain Generalization](https://mlsanity.com/paper/2104.09937/) (2021)
- [SelfReg: Self-supervised Contrastive Regularization for Domain Generalization](https://mlsanity.com/paper/2104.09841/) (2021)
- [DLOW: Domain Flow for Adaptation and Generalization](https://mlsanity.com/paper/1812.05418/) (2018)
- [Domain Adaptive Ensemble Learning](https://mlsanity.com/paper/2003.07325/) (2020)

## Built on by (most cited first, 29 of 29)

- [Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models](https://mlsanity.com/paper/2305.18455/) (2023)
- [Diverse Weight Averaging for Out-of-Distribution Generalization](https://mlsanity.com/paper/2205.09739/) (2022)
- [Sparse Mixture-of-Experts are Domain Generalizable Learners](https://mlsanity.com/paper/2206.04046/) (2022)
- [A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance](https://mlsanity.com/paper/2309.12530/) (2023)
- [Learning Domain Invariant Prompt for Vision-Language Models](https://mlsanity.com/paper/2212.04196/) (2022)
- [Domain-Unified Prompt Representations for Source-Free Domain Generalization](https://mlsanity.com/paper/2209.14926/) (2022)
- [Improving Domain Generalization with Domain Relations](https://mlsanity.com/paper/2302.02609/) (2023)
- [Explore and Exploit the Diverse Knowledge in Model Zoo for Domain Generalization](https://mlsanity.com/paper/2306.02595/) (2023)
- [QT-DoG: Quantization-aware Training for Domain Generalization](https://mlsanity.com/paper/2410.06020/) (2024)
- [Generalization in Neural Networks: A Broad Survey](https://mlsanity.com/paper/2209.01610/) (2022)
- [Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes](https://mlsanity.com/paper/2412.13573/) (2024)
- [Gradient Estimation for Unseen Domain Risk Minimization with Pre-Trained Models](https://mlsanity.com/paper/2302.01497/) (2023)
- [Prompt Disentanglement via Language Guidance and Representation Alignment for Domain Generalization](https://mlsanity.com/paper/2507.02288/) (2025)
- [Balanced Direction from Multifarious Choices: Arithmetic Meta-Learning for Domain Generalization](https://mlsanity.com/paper/2503.18987/) (2025)
- [LatentDR: Improving Model Generalization Through Sample-Aware Latent Degradation and Restoration](https://mlsanity.com/paper/2308.14596/) (2023)
- [CADG: A Model Based on Cross Attention for Domain Generalization](https://mlsanity.com/paper/2203.17067/) (2022)
- [Learning an Invertible Output Mapping Can Mitigate Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2210.01360/) (2022)
- [TDG: Text-guided Domain Generalization](https://mlsanity.com/paper/2308.09931/) (2023)
- [A Scene-aware Models Adaptation Scheme for Cross-scene Online Inference on Mobile Devices](https://mlsanity.com/paper/2407.03331/) (2024)
- [FedBaF: Federated Learning Aggregation Biased by a Foundation Model](https://mlsanity.com/paper/2410.18352/) (2024)
- [ShiftKD: Benchmarking Knowledge Distillation under Distribution Shift](https://mlsanity.com/paper/2312.16242/) (2023)
- [CoDEx: Combining Domain Expertise for Spatial Generalization in Satellite Image Analysis](https://mlsanity.com/paper/2504.19737/) (2025)
- [Memory-Integrated Reconfigurable Adapters: A Unified Framework for Settings with Multiple Tasks](https://mlsanity.com/paper/2512.00940/) (2025)
- [FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Model](https://mlsanity.com/paper/2606.20110/) (2026)
- [Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalization](https://mlsanity.com/paper/2311.15145/) (2023)
- [Hybrid Deep Learning-Based for Enhanced Occlusion Segmentation in PICU Patient Monitoring](https://mlsanity.com/paper/2407.13341/) (2024)
- [Collaborative Editable Model](https://mlsanity.com/paper/2506.14146/) (2025)
- [Deep neural networks with Fisher vector encoding for medical image classification](https://mlsanity.com/paper/2605.01667/) (2026)
- [Domain Generalization via Text-Anchored Information Bottleneck](https://mlsanity.com/paper/2607.01657/) (2026)

## Abstract (arXiv)

Fine-tuning pretrained models is a common practice in domain generalization (DG) tasks. However, fine-tuning is usually computationally expensive due to the ever-growing size of pretrained models. More importantly, it may cause over-fitting on source domain and compromise their generalization ability as shown in recent works. Generally, pretrained models possess some level of generalization ability and can achieve decent performance regarding specific domains and samples. However, the generalization performance of pretrained models could vary significantly over different test domains even samples, which raises challenges for us to best leverage pretrained models in DG tasks. In this paper, we propose a novel domain generalization paradigm to better leverage various pretrained models, named specialized ensemble learning for domain generalization (SEDGE). It first trains a linear label space adapter upon fixed pretrained models, which transforms the outputs of the pretrained model to the label space of the target domain. Then, an ensemble network aware of model specialty is proposed to dynamically dispatch proper pretrained models to predict each test sample. Experimental studies on several benchmarks show that SEDGE achieves significant performance improvements comparing to strong baselines including state-of-the-art method in DG tasks and reduces the trainable parameters by ~99% and the training time by ~99.5%.