mlsanityData to 7 Oct 2026

Exploring and Predicting Transferability across NLP Tasks

arXiv 2005.00770 ↗First posted 2 May 2020cs.CLcited by 126

Not on the map. Why?

Abstract (arXiv)

Recent advances in NLP demonstrate the effectiveness of training large-scale language models and transferring them to downstream tasks. Can fine-tuning these models on tasks other than language modeling further improve performance? In this paper, we conduct an extensive study of the transferability between 33 NLP tasks across three broad classes of problems (text classification, question answering, and sequence labeling). Our results show that transfer learning is more beneficial than previously thought, especially when target task data is scarce, and can improve performance even when the source task is small or differs substantially from the target task (e.g., part-of-speech tagging transfers well to the DROP QA dataset). We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task, and we validate their effectiveness in experiments controlled for source and target data size. Overall, our experiments reveal that factors such as source data size, task and domain similarity, and task complexity all play a role in determining transferability.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 33; 2023: 23; 2024: 24; 2025: 7; 2026: 11.

126 papers.

Sort:

By area:

  1. 2022Editing Models with Task Arithmeticcited by 1,076
  2. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  3. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  4. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  5. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  6. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  7. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  8. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  9. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  10. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  11. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  12. 2023Deep Model Fusion: A Surveycited by 89
38 more
  1. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  2. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  3. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 74
  4. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  5. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  6. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  7. 2021Comparing Test Sets with Item Response Theorycited by 55
  8. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  9. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
  10. 2022Memory Efficient Continual Learning with Transformerscited by 43
  11. 2022Unsupervised Cross-Task Generalization via Retrieval Augmentationcited by 42
  12. 2023π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolationcited by 40
  13. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  14. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasetscited by 36
  15. 2021Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responsescited by 34
  16. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  17. 2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learningcited by 33
  18. 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
  19. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  20. 2022Exploring Mode Connectivity for Pre-trained Language Modelscited by 26
  21. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
  22. 2023BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processingcited by 25
  23. 2021Predicting the Performance of Multilingual NLP Modelscited by 24
  24. 2022Where to start? Analyzing the potential value of intermediate modelscited by 23
  25. 2023Identification of Negative Transfers in Multitask Learning Using Surrogate Modelscited by 20
  26. 2024Scalable Multitask Learning Using Gradient-based Estimation of Task Affinitycited by 20
  27. 2023Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Datacited by 19
  28. 2021Rethinking Why Intermediate-Task Fine-Tuning Workscited by 18
  29. 2022Exploring the Role of Task Transferability in Large-Scale Multi-Task Learningcited by 17
  30. 2022When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learningcited by 17
  31. 2023Recyclable Tuning for Continual Pre-trainingcited by 17
  32. 2024SMART: Submodular Data Mixture Strategy for Instruction Tuningcited by 16
  33. 2023Boosting Multitask Learning on Graphs through Higher-Order Task Affinitiescited by 15
  34. 2022Geographic Adaptation of Pretrained Language Modelscited by 14
  35. 2024LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Modelscited by 14
  36. 2025LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selectioncited by 14
  37. 2023Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Datacited by 13
  38. 2024Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approachcited by 13

50 most cited of 126 All 126 (JSON)