mlsanityData to 7 Oct 2026

The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders

arXiv 2109.06939 ↗First posted 14 Sep 2021cs.CL cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

Multi-task learning with transformer encoders (MTL) has emerged as a powerful technique to improve performance on closely-related tasks for both accuracy and efficiency while a question still remains whether or not it would perform as well on tasks that are distinct in nature. We first present MTL results on five NLP tasks, POS, NER, DEP, CON, and SRL, and depict its deficiency over single-task learning. We then conduct an extensive pruning analysis to show that a certain set of attention heads get claimed by most tasks during MTL, who interfere with one another to fine-tune those heads for their own objectives. Based on this finding, we propose the Stem Cell Hypothesis to reveal the existence of attention heads naturally talented for many tasks that cannot be jointly trained to create adequate embeddings for all of those tasks. Finally, we design novel parameter-free probes to justify our hypothesis and demonstrate how attention heads are transformed across the five tasks during MTL through label analysis.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

24 of 38 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  9. 2019Designing and Interpreting Probes with Control Taskscited by 608
  10. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  11. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  12. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
12 more
  1. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  2. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  3. 2020Named Entity Recognition as Dependency Parsingcited by 150
  4. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  5. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  6. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  7. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  8. 2020Fast and Accurate Neural CRF Constituency Parsingcited by 50
  9. 2018Jointly Predicting Predicates and Arguments in Neural Semantic Role Labelingcited by 34
  10. 2019Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERTcited by 19
  11. 2019A Syntax-aware Multi-task Learning Framework for Chinese Semantic Role Labelingcited by 5
  12. 2020Relabel the Noise: Joint Extraction of Entities and Relations via Cooperative Multiagentscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 10; 2024: 7; 2025: 8; 2026: 2.

31 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  2. 2023Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Familycited by 71
  3. 2023Empirical Study of Zero-Shot NER with ChatGPTcited by 60
  4. 2024COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuningcited by 45
  5. 2023Fairness-Aware Structured Pruning in Transformerscited by 21
  6. 2022MVP: Multi-task Supervised Pre-training for Natural Language Generationcited by 16
  7. 2021Semantic Role Labeling as Dependency Parsing: Exploring Latent Tree Structures Inside Argumentscited by 14
  8. 2022CDConv: A Benchmark for Contradiction Detection in Chinese Conversationscited by 10
  9. 2022Improving Chinese Story Generation via Awareness of Syntactic Dependencies and Semanticscited by 10
  10. 2023InstructIE: A Bilingual Instruction-based Information Extraction Datasetcited by 9
  11. 2025A Cooperative Multi-Agent Framework for Zero-Shot Named Entity Recognitioncited by 5
  12. 2025Span-Level Hallucination Detection for LLM-Generated Answerscited by 5
19 more
  1. 2023Discourse Representation Structure Parsing for Chinesecited by 3
  2. 2023Audience-specific Explanations for Machine Translationcited by 3
  3. 2024An efficient text augmentation approach for contextualized Mandarin speech recognitioncited by 2
  4. 2025Structuring the Unstructured: A Systematic Review of Text-to-Structure Generation for Agentic AI with a Universal Evaluation Frameworkcited by 2
  5. 2025A Linguistics-Aware LLM Watermarking via Syntactic Predictabilitycited by 2
  6. 2023Unleashing the True Potential of Sequence-to-Sequence Models for Sequence Tagging and Structure Parsingcited by 1
  7. 2024HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistancecited by 1
  8. 2024Pre-training, Fine-tuning and Re-ranking: A Three-Stage Framework for Legal Question Answeringcited by 1
  9. 2023Friend-training: Learning from Models of Different but Related Taskscited by 0
  10. 2023CPPF: A contextual and post-processing-free model for automatic speech recognitioncited by 0
  11. 2023Rethinking Relation Classification with Graph Meaning Representationscited by 0
  12. 2024Labeling supervised fine-tuning data with the scaling lawcited by 0
  13. 2024Leveraging Entailment Judgements in Cross-Lingual Summarisationcited by 0
  14. 2025Student-Powered Digital Scholarship CoLab Project in the HKUST Library: Develop a Chinese Named-Entity Recognition (NER) Tool within One Semester from the Ground Upcited by 0
  15. 2025Weak Supervision Techniques towards Enhanced ASR Models in Industry-level CRM Systemscited by 0
  16. 2025Translation via Annotation: A Computational Study of Translating Classical Chinese into Japanesecited by 0
  17. 2025The Art of Storytelling in Authoritarian Regimes: Crafting State Narratives on Chinese Social Mediacited by 0
  18. 2026Towards the Holographic Characteristic of LLMs for Efficient Short-text Generationcited by 0
  19. 2026Linguistics-Aware Non-Distortionary LLM Watermarkingcited by 0