# Learning to select data for transfer learning with Bayesian Optimization

arXiv: https://arxiv.org/abs/1707.05246 · HTML: https://mlsanity.com/paper/1707.05246/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2017.

## Built on by (most cited first, 30 of 64)

- [Data Selection for Language Models via Importance Resampling](https://mlsanity.com/paper/2302.03169/) (2023)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models](https://mlsanity.com/paper/2307.14430/) (2023)
- [ClimateBert: A Pretrained Language Model for Climate-Related Text](https://mlsanity.com/paper/2110.12010/) (2021)
- [Continual Learning in the Teacher-Student Setup: Impact of Task Similarity](https://mlsanity.com/paper/2107.04384/) (2021)
- [Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training](https://mlsanity.com/paper/2504.13161/) (2025)
- [Multi-Source Domain Adaptation with Mixture of Experts](https://mlsanity.com/paper/1809.02256/) (2018)
- [ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation](https://mlsanity.com/paper/2306.04344/) (2023)
- [Uncertainty in Natural Language Processing: Sources, Quantification, and Applications](https://mlsanity.com/paper/2306.04459/) (2023)
- [Adversarial Learning for Zero-Shot Stance Detection on Social Media](https://mlsanity.com/paper/2105.06603/) (2021)
- [Efficient Continual Pre-training for Building Domain Specific Large Language Models](https://mlsanity.com/paper/2311.08545/) (2023)
- [AdaPrompt: Adaptive Model Training for Prompt-based NLP](https://mlsanity.com/paper/2202.04824/) (2022)
- [Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation](https://mlsanity.com/paper/2312.12480/) (2023)
- [Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-Training](https://mlsanity.com/paper/2109.06050/) (2021)
- [TSDS: Data Selection for Task-Specific Model Finetuning](https://mlsanity.com/paper/2410.11303/) (2024)
- [Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts](https://mlsanity.com/paper/2402.07625/) (2024)
- [Checkpoint Merging via Bayesian Optimization in LLM Pretraining](https://mlsanity.com/paper/2403.19390/) (2024)
- [Focus on the Common Good: Group Distributional Robustness Follows](https://mlsanity.com/paper/2110.02619/) (2021)
- [Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling](https://mlsanity.com/paper/2410.03735/) (2024)
- [The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures](https://mlsanity.com/paper/2104.10640/) (2021)
- [Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization](https://mlsanity.com/paper/2109.06471/) (2021)
- [Synthetic Question Value Estimation for Domain Adaptation of Question Answering](https://mlsanity.com/paper/2203.08926/) (2022)
- [Dynamic Fisher-weighted Model Merging via Bayesian Optimization](https://mlsanity.com/paper/2504.18992/) (2025)
- [To Share or not to Share: Predicting Sets of Sources for Model Transfer Learning](https://mlsanity.com/paper/2104.08078/) (2021)
- [Evaluating the Values of Sources in Transfer Learning](https://mlsanity.com/paper/2104.12567/) (2021)
- [Active Learning Over Multiple Domains in Natural Language Tasks](https://mlsanity.com/paper/2202.00254/) (2022)
- [Automatic Document Selection for Efficient Encoder Pretraining](https://mlsanity.com/paper/2210.10951/) (2022)
- [NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysis](https://mlsanity.com/paper/2305.00090/) (2023)
- [Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptation](https://mlsanity.com/paper/2405.16486/) (2024)
- [Learning Visual Models using a Knowledge Graph as a Trainer](https://mlsanity.com/paper/2102.08747/) (2021)

## Abstract (arXiv)

Domain similarity measures can be used to gauge adaptability and select suitable data for transfer learning, but existing approaches define ad hoc measures that are deemed suitable for respective tasks. Inspired by work on curriculum learning, we propose to learn data selection measures using Bayesian Optimization and evaluate them across models, domains and tasks. Our learned measures outperform existing domain similarity measures significantly on three tasks: sentiment analysis, part-of-speech tagging, and parsing. We show the importance of complementing similarity with diversity, and that learned measures are -- to some degree -- transferable across models, domains, and even tasks.