# To Share or not to Share: Predicting Sets of Sources for Model Transfer Learning

arXiv: https://arxiv.org/abs/2104.08078 · HTML: https://mlsanity.com/paper/2104.08078/

## Key facts

- Cited by 11 papers in the map (data through 13 Oct 2026).
- Builds on 16 of its 60 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://mlsanity.com/paper/2010.12309/) (2020)
- [Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?](https://mlsanity.com/paper/2005.00628/) (2020)
- [Exploring and Predicting Transferability across NLP Tasks](https://mlsanity.com/paper/2005.00770/) (2020)
- [Learning to select data for transfer learning with Bayesian Optimization](https://mlsanity.com/paper/1707.05246/) (2017)
- [Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification](https://mlsanity.com/paper/1908.11860/) (2019)
- [Identifying beneficial task relations for multi-task learning in deep neural networks](https://mlsanity.com/paper/1702.08303/) (2017)
- [The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domain](https://mlsanity.com/paper/2006.03039/) (2020)
- [Generalizing Natural Language Analysis through Span-relation Representations](https://mlsanity.com/paper/1911.03822/) (2019)
- [Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media](https://mlsanity.com/paper/2010.01150/) (2020)
- [Using Similarity Measures to Select Pretraining Data for NER](https://mlsanity.com/paper/1904.00585/) (2019)
- [Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Text](https://mlsanity.com/paper/2005.09392/) (2020)
- [Multi-task Domain Adaptation for Sequence Tagging](https://mlsanity.com/paper/1608.02689/) (2016)
- [WNUT-2020 Task 1 Overview: Extracting Entities and Relations from Wet Lab Protocols](https://mlsanity.com/paper/2010.14576/) (2020)

## Built on by (most cited first, 11 of 11)

- [Unsupervised Cross-Task Generalization via Retrieval Augmentation](https://mlsanity.com/paper/2204.07937/) (2022)
- [A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing](https://mlsanity.com/paper/2210.04675/) (2022)
- [NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysis](https://mlsanity.com/paper/2305.00090/) (2023)
- [CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain](https://mlsanity.com/paper/2112.08754/) (2021)
- [TADA: Efficient Task-Agnostic Domain Adaptation for Transformers](https://mlsanity.com/paper/2305.12717/) (2023)
- [GradSim: Gradient-Based Language Grouping for Effective Multilingual Training](https://mlsanity.com/paper/2310.15269/) (2023)
- [AAdaM at SemEval-2024 Task 1: Augmentation and Adaptation for Multilingual Semantic Textual Relatedness](https://mlsanity.com/paper/2404.01490/) (2024)
- [Selecting Subsets of Source Data for Transfer Learning with Applications in Metal Additive Manufacturing](https://mlsanity.com/paper/2401.08715/) (2024)
- [Leveraging Open Data and Task Augmentation to Automated Behavioral Coding of Psychotherapy Conversations in Low-Resource Scenarios](https://mlsanity.com/paper/2210.14254/) (2022)
- [Boosting Transformers for Job Expression Extraction and Classification in a Low-Resource Setting](https://mlsanity.com/paper/2109.08597/) (2021)
- [Statistical Deficiency for Task Inclusion Estimation](https://mlsanity.com/paper/2503.05491/) (2025)

## Abstract (arXiv)

In low-resource settings, model transfer can help to overcome a lack of labeled data for many tasks and domains. However, predicting useful transfer sources is a challenging problem, as even the most similar sources might lead to unexpected negative transfer results. Thus, ranking methods based on task and text similarity -- as suggested in prior work -- may not be sufficient to identify promising sources. To tackle this problem, we propose a new approach to automatically determine which and how many sources should be exploited. For this, we study the effects of model transfer on sequence labeling across various domains and tasks and show that our methods based on model similarity and support vector machines are able to predict promising sources, resulting in performance increases of up to 24 F1 points.