mlsanityData to 14 Oct 2026

Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning

arXiv 2204.11117 ↗First posted 23 Apr 2022cs.CL cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. In contrast, literature on task transferability has established that the choice of intermediate tasks can heavily affect downstream task performance. In this work, we aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning. We find that, on average, increasing the scale of multi-task learning, in terms of the number of tasks, indeed results in better learned representations than smaller multi-task setups. However, if the target tasks are known ahead of time, then training on a smaller set of related tasks is competitive to the large-scale multi-task training at a reduced computational cost.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 27 of its 46 references that are in the map. First posted on arXiv 23 Apr 2022.

Builds on

27 of 46 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  10. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  11. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  12. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
15 more
  1. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
  2. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  3. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  4. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  5. 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
  6. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  7. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  8. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  9. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 157
  10. 2021Few-NERD: A Few-Shot Named Entity Recognition Datasetcited by 152
  11. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  12. 2020Exploring and Predicting Transferability across NLP Taskscited by 128
  13. 2016Many Languages, One Parsercited by 26
  14. 2018Scheduled Multi-Task Learning: From Syntax to Translationcited by 18
  15. 2018Multi-Task Learning for Sequence Tagging: An Empirical Studycited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 6; 2024: 1; 2025: 3; 2026: 1.

17 papers.

Sort:

  1. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  2. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 129
  3. 2022Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writingcited by 71
  4. 2023Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmarkcited by 63
  5. 2022Task Compass: Scaling Multi-task Pre-training with Task Prefixcited by 14
  6. 2023The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarkscited by 12
  7. 2023TaskWeb: Selecting Better Source Tasks for Multi-task NLPcited by 11
  8. 2023Understanding the Transferability of Representations via Task-Relatednesscited by 11
  9. 2022Evidence > Intuition: Transferability Estimation for Encoder Selectioncited by 7
  10. 2022Few-shot Adaptation Works with UnpredicTable Datacited by 6
  11. 2023When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLPcited by 5
  12. 2023Pre-Finetuning for Few-Shot Emotional Speech Recognitioncited by 4
5 more
  1. 2026Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Taskscited by 4
  2. 2025Improving Dialectal Slot and Intent Detection with Auxiliary Tasks: A Multi-Dialectal Bavarian Case Studycited by 3
  3. 2025Add Noise, Tasks, or Layers? MaiNLP at the VarDial 2025 Shared Task on Norwegian Dialectal Slot and Intent Detectioncited by 2
  4. 2025Latent Traits and Cross-Task Transfer: Deconstructing Dataset Interactions in LLM Fine-tuningcited by 2
  5. 2024Linguistic Knowledge Can Enhance Encoder-Decoder Models (If You Let It)cited by 0