mlsanityData to 7 Oct 2026

Scalable Transfer Learning with Expert Models

arXiv 2009.13239 ↗First posted 28 Sep 2020cs.LG cs.CV stat.MLcited by 48

Not on the map. Why?

Abstract (arXiv)

Transfer of pre-trained representations can improve sample efficiency and reduce computational requirements for new tasks. However, representations used for transfer are usually generic, and are not tailored to a particular distribution of downstream tasks. We explore the use of expert representations for transfer with a simple, yet effective, strategy. We train a diverse set of experts by exploiting existing label structures, and use cheap-to-compute performance proxies to select the relevant expert for each target task. This strategy scales the process of transferring to new tasks, since it does not revisit the pre-training data during transfer. Accordingly, it requires little extra compute per target task, and results in a speed-up of 2-3 orders of magnitude compared to competing approaches. Further, we provide an adapter-based architecture able to compress many experts into a single model. We evaluate our approach on two different data sources and demonstrate that it outperforms baselines on over 20 diverse vision tasks in both cases.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 14; 2023: 7; 2024: 10; 2025: 4; 2026: 1.

48 papers.

Sort:

By area:

  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  3. 2021ImageNet-21K Pretraining for the Massescited by 607
  4. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  5. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  6. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  7. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  8. 2021Ensembling Off-the-shelf Models for GAN Trainingcited by 84
  9. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  10. 2021Self-Supervised Pretraining Improves Self-Supervised Pretrainingcited by 81
  11. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  12. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
36 more
  1. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  2. 2021Learning a Universal Template for Few-shot Dataset Generalizationcited by 61
  3. 2021Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Typescited by 54
  4. 2022Memory Efficient Continual Learning with Transformerscited by 43
  5. 2021Scalable Diverse Model Selection for Accessible Transfer Learningcited by 40
  6. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  7. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  8. 2021Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapterscited by 23
  9. 2022Proper Reuse of Image Classification Features Improves Object Detectioncited by 20
  10. 2022SphereFed: Hyperspherical Federated Learningcited by 19
  11. 2024LocMoE: A Low-Overhead MoE for Large Language Model Trainingcited by 18
  12. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Expertscited by 17
  13. 2024Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Expertscited by 16
  14. 2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
  15. 2022Deep transfer learning for image classification: a surveycited by 13
  16. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
  17. 2023Revisiting Single-gated Mixtures of Expertscited by 10
  18. 2024Conditional computation in neural networks: principles and research trendscited by 10
  19. 2022Granularity-aware Adaptation for Image Retrieval over Multiple Taskscited by 8
  20. 2022Content-Based Search for Deep Generative Modelscited by 7
  21. 2023Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settingscited by 7
  22. 2024Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Modelscited by 6
  23. 2021Representation Consolidation for Training Expert Studentscited by 5
  24. 2022Modularized Transfer Learning with Multiple Knowledge Graphs for Zero-shot Commonsense Reasoningcited by 5
  25. 2024Data Selection for Transfer Unlearningcited by 5
  26. 2022CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrationscited by 4
  27. 2023How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Surveycited by 4
  28. 2022SHiFT: An Efficient, Flexible Search Engine for Transfer Learningcited by 3
  29. 2024On Parameter Estimation in Deviated Gaussian Mixture of Expertscited by 2
  30. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  31. 2021Sequential Random Network for Fine-grained Image Classificationcited by 0
  32. 2023Direct Neural Machine Translation with Task-level Mixture of Experts modelscited by 0
  33. 2024Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Rankingcited by 0
  34. 2025How to Select Pre-Trained Code Models for Reuse? A Learning Perspectivecited by 0
  35. 2025MoMa: A Modular Deep Learning Framework for Material Property Predictioncited by 0
  36. 2026MoECodec: Image Compression for joint human and machine perception via Mixture-of-Expertscited by 0