mlsanityData to 7 Oct 2026

To Share or not to Share: Predicting Sets of Sources for Model Transfer Learning

arXiv 2104.08078 ↗First posted 16 Apr 2021cs.CL cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

In low-resource settings, model transfer can help to overcome a lack of labeled data for many tasks and domains. However, predicting useful transfer sources is a challenging problem, as even the most similar sources might lead to unexpected negative transfer results. Thus, ranking methods based on task and text similarity -- as suggested in prior work -- may not be sufficient to identify promising sources. To tackle this problem, we propose a new approach to automatically determine which and how many sources should be exploited. For this, we study the effects of model transfer on sequence labeling across various domains and tasks and show that our methods based on model similarity and support vector machines are able to predict promising sources, resulting in performance increases of up to 24 F1 points.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 16 of its 60 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

16 of 60 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  4. 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 170
  5. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  6. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  7. 2017Learning to select data for transfer learning with Bayesian Optimizationcited by 64
  8. 2019Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classificationcited by 62
  9. 2017Identifying beneficial task relations for multi-task learning in deep neural networkscited by 59
  10. 2020The SOFC-Exp Corpus and Neural Approaches to Information Extraction in the Materials Science Domaincited by 32
  11. 2019Generalizing Natural Language Analysis through Span-relation Representationscited by 22
  12. 2020Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Mediacited by 16
4 more
  1. 2019Using Similarity Measures to Select Pretraining Data for NERcited by 15
  2. 2020Adversarial Alignment of Multilingual Models for Extracting Temporal Expressions from Textcited by 11
  3. 2016Multi-task Domain Adaptation for Sequence Taggingcited by 8
  4. 2020WNUT-2020 Task 1 Overview: Extracting Entities and Relations from Wet Lab Protocolscited by 5

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 3; 2023: 3; 2024: 2; 2025: 1.

11 papers.

  1. 2022Unsupervised Cross-Task Generalization via Retrieval Augmentationcited by 42
  2. 2022A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processingcited by 20
  3. 2023NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysiscited by 10
  4. 2021CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domaincited by 7
  5. 2023TADA: Efficient Task-Agnostic Domain Adaptation for Transformerscited by 6
  6. 2023GradSim: Gradient-Based Language Grouping for Effective Multilingual Trainingcited by 6
  7. 2024AAdaM at SemEval-2024 Task 1: Augmentation and Adaptation for Multilingual Semantic Textual Relatednesscited by 4
  8. 2024Selecting Subsets of Source Data for Transfer Learning with Applications in Metal Additive Manufacturingcited by 3
  9. 2022Leveraging Open Data and Task Augmentation to Automated Behavioral Coding of Psychotherapy Conversations in Low-Resource Scenarioscited by 2
  10. 2021Boosting Transformers for Job Expression Extraction and Classification in a Low-Resource Settingcited by 1
  11. 2025Statistical Deficiency for Task Inclusion Estimationcited by 1