mlsanityData to 14 Oct 2026

Do Text-to-Text Multi-Task Learners Suffer from Task Conflict?

arXiv 2212.06645 ↗First posted 13 Dec 2022cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Traditional multi-task learning architectures train a single model across multiple tasks through a shared encoder followed by task-specific decoders. Learning these models often requires specialized training algorithms that address task-conflict in the shared parameter updates, which otherwise can lead to negative transfer. A new type of multi-task learning within NLP homogenizes multi-task architectures as a shared encoder and language model decoder, which does surprisingly well across a range of diverse tasks. Does this new architecture suffer from task-conflicts that require specialized training algorithms? We study how certain factors in the shift towards text-to-text models affects multi-task conflict and negative transfer, finding that both directional conflict and transfer are surprisingly constant across architectures.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 13 Dec 2022.

Builds on

28 of 41 references.

References · 28

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  7. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  9. 2018Neural Network Acceptability Judgmentscited by 958
  10. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  11. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  12. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
16 more
  1. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  2. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  3. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  4. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  5. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426
  6. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  7. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  8. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 236
  9. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 222
  10. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 201
  11. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 169
  12. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  13. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  14. 2021RotoGrad: Gradient Homogenization in Multitask Learningcited by 83
  15. 2022Auto-Lambda: Disentangling Dynamic Task Relationshipscited by 73
  16. 2022Do Current Multi-Task Optimization Methods in Deep Learning Even Help?cited by 60

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 2.

6 papers.

  1. 2024Amuro and Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Modelscited by 9
  2. 2023When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLPcited by 5
  3. 2025IL-PCSR: Legal Corpus for Prior Case and Statute Retrievalcited by 5
  4. 2025SEE: Continual Fine-tuning with Sequential Ensemble of Expertscited by 4
  5. 2024Are Clinical T5 Models Better for Clinical Text?cited by 3
  6. 2024Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learningcited by 0