# Explore and Exploit the Diverse Knowledge in Model Zoo for Domain Generalization

arXiv: https://arxiv.org/abs/2306.02595 · HTML: https://mlsanity.com/paper/2306.02595/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 68 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Self-Supervised Learning of Pretext-Invariant Representations](https://mlsanity.com/paper/1912.01991/) (2019)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)

## Built on by (most cited first, 14 of 14)

- [Spurious Feature Diversification Improves Out-of-distribution Generalization](https://mlsanity.com/paper/2309.17230/) (2023)
- [Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generation](https://mlsanity.com/paper/2503.13070/) (2025)
- [Elucidating The Design Space of Classifier-Guided Diffusion Generation](https://mlsanity.com/paper/2310.11311/) (2023)
- [HYPO: Hyperspherical Out-of-Distribution Generalization](https://mlsanity.com/paper/2402.07785/) (2024)
- [HCVP: Leveraging Hierarchical Contrastive Visual Prompt for Domain Generalization](https://mlsanity.com/paper/2401.09716/) (2024)
- [Out-of-Distribution Learning with Human Feedback](https://mlsanity.com/paper/2408.07772/) (2024)
- [Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion](https://mlsanity.com/paper/2402.16305/) (2024)
- [Selection, Ensemble, and Adaptation: Advancing Multi-Source-Free Domain Adaptation via Architecture Zoo](https://mlsanity.com/paper/2403.01582/) (2024)
- [Understanding Cross-Model Perceptual Invariances Through Ensemble Metamers](https://mlsanity.com/paper/2504.01739/) (2025)
- [Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights](https://mlsanity.com/paper/2501.03782/) (2025)
- [High-Rate Mixout: Revisiting Mixout for Robust Domain Generalization](https://mlsanity.com/paper/2510.06955/) (2025)
- [Symmetric Divergence and Normalized Similarity: A Unified Topological Framework for Representation Analysis](https://mlsanity.com/paper/2606.06342/) (2026)
- [Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing](https://mlsanity.com/paper/2607.09143/) (2026)
- [When More Foundation Models Means Less: Diagnosing and Addressing Multi-View Fusion Failure](https://mlsanity.com/paper/2608.17490/) (2026)

## Abstract (arXiv)

The proliferation of pretrained models, as a result of advancements in pretraining techniques, has led to the emergence of a vast zoo of publicly available models. Effectively utilizing these resources to obtain models with robust out-of-distribution generalization capabilities for downstream tasks has become a crucial area of research. Previous research has primarily focused on identifying the most powerful models within the model zoo, neglecting to fully leverage the diverse inductive biases contained within. This paper argues that the knowledge contained in weaker models is valuable and presents a method for leveraging the diversity within the model zoo to improve out-of-distribution generalization capabilities. Specifically, we investigate the behaviors of various pretrained models across different domains of downstream tasks by characterizing the variations in their encoded representations in terms of two dimensions: diversity shift and correlation shift. This characterization enables us to propose a new algorithm for integrating diverse pretrained models, not limited to the strongest models, in order to achieve enhanced out-of-distribution generalization performance. Our proposed method demonstrates state-of-the-art empirical results on a variety of datasets, thus validating the benefits of utilizing diverse knowledge.