# Domain Adaptive Transfer Learning with Specialist Models

arXiv: https://arxiv.org/abs/1811.07056 · HTML: https://mlsanity.com/paper/1811.07056/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2018.

## Built on by (most cited first, 30 of 31)

- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data](https://mlsanity.com/paper/2101.12037/) (2021)
- [Exploring the Limits of Large Scale Pre-training](https://mlsanity.com/paper/2110.02095/) (2021)
- [Self-Supervised Pretraining Improves Self-Supervised Pretraining](https://mlsanity.com/paper/2103.12718/) (2021)
- [What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factors](https://mlsanity.com/paper/2203.01825/) (2022)
- [Transferability in Deep Learning: A Survey](https://mlsanity.com/paper/2201.05867/) (2022)
- [PRIOR: Prototype Representation Joint Learning from Medical Images and Reports](https://mlsanity.com/paper/2307.12577/) (2023)
- [Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Types](https://mlsanity.com/paper/2103.13318/) (2021)
- [Transferability Estimation using Bhattacharyya Class Separability](https://mlsanity.com/paper/2111.12780/) (2021)
- [Current state of LLM Risks and AI Guardrails](https://mlsanity.com/paper/2406.12934/) (2024)
- [Adaptive Consistency Regularization for Semi-Supervised Transfer Learning](https://mlsanity.com/paper/2103.02193/) (2021)
- [A Data-Based Perspective on Transfer Learning](https://mlsanity.com/paper/2207.05739/) (2022)
- [How stable are Transferability Metrics evaluations?](https://mlsanity.com/paper/2204.01403/) (2022)
- [Transferability Metrics for Selecting Source Model Ensembles](https://mlsanity.com/paper/2111.13011/) (2021)
- [Auxiliary Task Update Decomposition: The Good, The Bad and The Neutral](https://mlsanity.com/paper/2108.11346/) (2021)
- [Deep transfer learning for image classification: a survey](https://mlsanity.com/paper/2205.09904/) (2022)
- [Adaptive Training Distributions with Scalable Online Bilevel Optimization](https://mlsanity.com/paper/2311.11973/) (2023)
- [Fine-Grained Adversarial Semi-supervised Learning](https://mlsanity.com/paper/2110.05848/) (2021)
- [Scalable Neural Data Server: A Data Recommender for Transfer Learning](https://mlsanity.com/paper/2206.09386/) (2022)
- [ScaleNet: Searching for the Model to Scale](https://mlsanity.com/paper/2207.07267/) (2022)
- [Fairness-Aware Data Valuation for Supervised Learning](https://mlsanity.com/paper/2303.16963/) (2023)
- [Towards Understanding the Effect of Pretraining Label Granularity](https://mlsanity.com/paper/2303.16887/) (2023)
- [A Bayesian Approach to Data Point Selection](https://mlsanity.com/paper/2411.03768/) (2024)
- [Non-binary deep transfer learning for image classification](https://mlsanity.com/paper/2107.08585/) (2021)
- [SEPT: Towards Scalable and Efficient Visual Pre-Training](https://mlsanity.com/paper/2212.05473/) (2022)
- [Mixture Data for Training Cannot Ensure Out-of-distribution Generalization](https://mlsanity.com/paper/2312.16243/) (2023)
- [Sequential Random Network for Fine-grained Image Classification](https://mlsanity.com/paper/2103.07230/) (2021)
- [Fighting Selection Bias in Statistical Learning: Application to Visual Recognition from Biased Image Databases](https://mlsanity.com/paper/2109.02357/) (2021)
- [Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search](https://mlsanity.com/paper/2112.00275/) (2021)
- [ISTRBoost: Importance Sampling Transfer Regression using Boosting](https://mlsanity.com/paper/2204.12044/) (2022)

## Abstract (arXiv)

Transfer learning is a widely used method to build high performing computer vision models. In this paper, we study the efficacy of transfer learning by examining how the choice of data impacts performance. We find that more pre-training data does not always help, and transfer performance depends on a judicious choice of pre-training data. These findings are important given the continued increase in dataset sizes. We further propose domain adaptive transfer learning, a simple and effective pre-training method using importance weights computed based on the target dataset. Our method to compute importance weights follow from ideas in domain adaptation, and we show a novel application to transfer learning. Our methods achieve state-of-the-art results on multiple fine-grained classification datasets and are well-suited for use in practice.