mlsanityData to 7 Oct 2026

TaskWeb: Selecting Better Source Tasks for Multi-task NLP

arXiv 2305.13256 ↗First posted 22 May 2023cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent work in NLP has shown promising results in training models on large amounts of tasks to achieve better generalization. However, it is not well-understood how tasks are related, and how helpful training tasks can be chosen for a new task. In this work, we investigate whether knowing task relationships via pairwise task transfer improves choosing one or more source tasks that help to learn a new target task. We provide TaskWeb, a large-scale benchmark of pairwise task transfers for 22 NLP tasks using three different model types, sizes, and adaptation methods, spanning about 25,000 experiments. Then, we design a new method TaskShop based on our analysis of TaskWeb. TaskShop uses TaskWeb to estimate the benefit of using a source task for learning a new target task, and to choose a subset of helpful training tasks for multi-task training. Our method improves overall rankings and top-k precision of source tasks by 10% and 38%, respectively. We also use TaskShop to build much smaller multi-task training sets that improve zero-shot performances across 11 different target tasks by at least 4.3%.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 46 of its 70 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

46 of 70 references.

References · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
34 more
  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  6. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  7. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  8. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  9. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  10. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  11. 2021MetaICL: Learning to Learn In Contextcited by 450
  12. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  13. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  14. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  15. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  16. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  17. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  18. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  19. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  20. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  21. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  22. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  23. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 120
  24. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  25. 2019QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questionscited by 89
  26. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  27. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  28. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 74
  29. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  30. 2022Unsupervised Cross-Task Generalization via Retrieval Augmentationcited by 42
  31. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
  32. 2022Exploring the Role of Task Transferability in Large-Scale Multi-Task Learningcited by 17
  33. 2022FETA: A Benchmark for Few-Sample Task Transfer in Open-Domain Dialoguecited by 11
  34. 2022Few-shot Adaptation Works with UnpredicTable Datacited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 0; 2026: 2.

11 papers.

  1. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  2. 2023CITB: A Benchmark for Continual Instruction Tuningcited by 32
  3. 2023BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processingcited by 25
  4. 2024CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitionscited by 13
  5. 2024Instruction Matters: A Simple yet Effective Task Selection for Optimized Instruction Tuning of Specific Taskscited by 6
  6. 2024The Right Time Matters: Data Arrangement Affects Zero-Shot Generalization in Instruction Tuningcited by 4
  7. 2024Instruction Embedding: Latent Representations of Instructions Towards Task Identificationcited by 4
  8. 2026Will it Merge? On The Causes of Model Mergeabilitycited by 3
  9. 2024Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learningcited by 2
  10. 2024Back to the Basics on Predicting Transfer Performancecited by 0
  11. 2026A helps B while B hurts A: directed transfer in instruction-tuning mixturecited by 0