# Zoo-Tuning: Adaptive Transfer from a Zoo of Models

arXiv: https://arxiv.org/abs/2106.15434 · HTML: https://mlsanity.com/paper/2106.15434/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 60 references that are in the map.
- First posted on arXiv 29 Jun 2021.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark](https://mlsanity.com/paper/1910.04867/) (2019)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [A Survey on Deep Transfer Learning](https://mlsanity.com/paper/1808.01974/) (2018)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks](https://mlsanity.com/paper/1406.6909/) (2014)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [High-Resolution Representations for Labeling Pixels and Regions](https://mlsanity.com/paper/1904.04514/) (2019)
- [LEEP: A New Measure to Evaluate Transferability of Learned Representations](https://mlsanity.com/paper/2002.12462/) (2020)
- [LogME: Practical Assessment of Pre-trained Models for Transfer Learning](https://mlsanity.com/paper/2102.11005/) (2021)

## Built on by (most cited first, 30 of 35)

- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Deep Model Fusion: A Survey](https://mlsanity.com/paper/2309.15698/) (2023)
- [Deep Model Reassembly](https://mlsanity.com/paper/2210.17409/) (2022)
- [Hyper-Representations as Generative Models: Sampling Unseen Neural Network Weights](https://mlsanity.com/paper/2209.14733/) (2022)
- [Transferability in Deep Learning: A Survey](https://mlsanity.com/paper/2201.05867/) (2022)
- [Model Zoos: A Dataset of Diverse Populations of Neural Network Models](https://mlsanity.com/paper/2209.14764/) (2022)
- [Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs](https://mlsanity.com/paper/2110.10545/) (2021)
- [Adversarial Robustness for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2109.00946/) (2021)
- [Transferability Metrics for Selecting Source Model Ensembles](https://mlsanity.com/paper/2111.13011/) (2021)
- [FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models](https://mlsanity.com/paper/2401.15636/) (2024)
- [Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How](https://mlsanity.com/paper/2306.03828/) (2023)
- [Elucidating The Design Space of Classifier-Guided Diffusion Generation](https://mlsanity.com/paper/2310.11311/) (2023)
- [ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization](https://mlsanity.com/paper/2210.09236/) (2022)
- [Towards Instance-adaptive Inference for Federated Learning](https://mlsanity.com/paper/2308.06051/) (2023)
- [Agile Multi-Source-Free Domain Adaptation](https://mlsanity.com/paper/2403.05062/) (2024)
- [Transferring Knowledge from Large Foundation Models to Small Downstream Models](https://mlsanity.com/paper/2406.07337/) (2024)
- [Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/2205.14410/) (2022)
- [Hub-Pathway: Transfer Learning from A Hub of Pre-trained Models](https://mlsanity.com/paper/2206.03726/) (2022)
- [Transfer Learning with Pre-trained Conditional Generative Models](https://mlsanity.com/paper/2204.12833/) (2022)
- [Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Models](https://mlsanity.com/paper/2304.13718/) (2023)
- [Efficient Stitchable Task Adaptation](https://mlsanity.com/paper/2311.17352/) (2023)
- [Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion](https://mlsanity.com/paper/2402.16305/) (2024)
- [Provable Pathways: Learning Multiple Tasks over Multiple Paths](https://mlsanity.com/paper/2303.04338/) (2023)
- [Federated Adversarial Learning for Robust Autonomous Landing Runway Detection](https://mlsanity.com/paper/2406.15925/) (2024)
- [Instruction-Guided Autoregressive Neural Network Parameter Generation](https://mlsanity.com/paper/2504.02012/) (2025)
- [A Model Zoo on Phase Transitions in Neural Networks](https://mlsanity.com/paper/2504.18072/) (2025)
- [Transfer Learning via Test-Time Neural Networks Aggregation](https://mlsanity.com/paper/2206.13399/) (2022)
- [Beyond Adapter Retrieval: Latent Geometry-Preserving Composition via Sparse Task Projection](https://mlsanity.com/paper/2410.09908/) (2024)
- [Selective Cross-Task Distillation](https://mlsanity.com/paper/2204.11526/) (2022)
- [ViM: Vision Middleware for Unified Downstream Transferring](https://mlsanity.com/paper/2303.06911/) (2023)

## Abstract (arXiv)

With the development of deep networks on various large-scale datasets, a large zoo of pretrained models are available. When transferring from a model zoo, applying classic single-model based transfer learning methods to each source model suffers from high computational burden and cannot fully utilize the rich knowledge in the zoo. We propose Zoo-Tuning to address these challenges, which learns to adaptively transfer the parameters of pretrained models to the target task. With the learnable channel alignment layer and adaptive aggregation layer, Zoo-Tuning adaptively aggregates channel aligned pretrained parameters to derive the target model, which promotes knowledge transfer by simultaneously adapting multiple source models to downstream tasks. The adaptive aggregation substantially reduces the computation cost at both training and inference. We further propose lite Zoo-Tuning with the temporal ensemble of batch average gating values to reduce the storage cost at the inference time. We evaluate our approach on a variety of tasks, including reinforcement learning, image classification, and facial landmark detection. Experiment results demonstrate that the proposed adaptive transfer learning approach can transfer knowledge from a zoo of models more effectively and efficiently.