mlsanityData to 7 Oct 2026

A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods

arXiv 2204.03508 ↗First posted 7 Apr 2022cs.CL cs.AIcited by 42

Not on the map. Why?

Abstract (arXiv)

Multi-task learning (MTL) has become increasingly popular in natural language processing (NLP) because it improves the performance of related tasks by exploiting their commonalities and differences. Nevertheless, it is still not understood very well how multi-task learning can be implemented based on the relatedness of training tasks. In this survey, we review recent advances of multi-task learning methods in NLP, with the aim of summarizing them into two general multi-task training methods based on their task relatedness: (i) joint training and (ii) multi-step training. We present examples in various NLP downstream applications, summarize the task relationships and discuss future directions of this promising topic.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 75 of its 108 references that are in the map. First posted on arXiv 7 Apr 2022.

Builds on

75 of 108 references.

References · 75

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  9. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  10. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  11. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  12. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
38 more
  1. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  2. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  3. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  4. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  5. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  6. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
  7. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  8. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  9. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  10. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  11. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  12. 2016Recurrent Neural Network for Text Classification with Multi-Task Learningcited by 144
  13. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 143
  14. 2016Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Fillingcited by 137
  15. 2020A Survey of Knowledge-Enhanced Text Generationcited by 119
  16. 2019A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understandingcited by 106
  17. 2020Entities as Experts: Sparse Memory Access with Entity Supervisioncited by 100
  18. 2015Multi-task Sequence to Sequence Learningcited by 94
  19. 2017Adversarial Multi-task Learning for Text Classificationcited by 91
  20. 2019Proactive Human-Machine Conversation with Explicit Conversation Goalscited by 87
  21. 2020Sequential Latent Knowledge Selection for Knowledge-Grounded Dialoguecited by 82
  22. 2018Semi-Supervised Sequence Modeling with Cross-View Trainingcited by 75
  23. 2018Joint Slot Filling and Intent Detection via Capsule Neural Networkscited by 73
  24. 2020A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generationcited by 72
  25. 2021KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answeringcited by 66
  26. 2019NCLS: Neural Cross-Lingual Summarizationcited by 64
  27. 2016A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Taskscited by 62
  28. 2020A Co-Interactive Transformer for Joint Slot Filling and Intent Detectioncited by 61
  29. 2018A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Taskscited by 54
  30. 2019Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphscited by 53
  31. 2021A Survey on Spoken Language Understanding: Recent Advances and New Frontierscited by 53
  32. 2021Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptationcited by 50
  33. 2021From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understandingcited by 43
  34. 2021Dict-BERT: Enhancing Language Model Pre-training with Dictionarycited by 38
  35. 2017Question Answering and Question Generation as Dual Taskscited by 37
  36. 2019CM-Net: A Novel Collaborative Memory Network for Spoken Language Understandingcited by 34
  37. 2019Multi-Task Learning for Conversational Question Answering over a Large-Scale Knowledge Basecited by 33
  38. 2021GL-GIN: Fast and Accurate Non-Autoregressive Model for Joint Multiple Intent Detection and Slot Fillingcited by 33

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 11; 2024: 16; 2025: 8; 2026: 5.

42 papers.

Sort:

By area:

  1. 2024Fair Resource Allocation in Multi-Task Learningcited by 34
  2. 2023MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Followingcited by 30
  3. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 29
  4. 2023Large Language Models are Built-in Autoregressive Search Enginescited by 26
  5. 2023The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generationcited by 20
  6. 2024MT2ST: Adaptive Multi-Task to Single-Task Learningcited by 16
  7. 2024Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Modelscited by 15
  8. 2022Task Compass: Scaling Multi-task Pre-training with Task Prefixcited by 14
  9. 2024Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approachcited by 13
  10. 2024CoBa: Convergence Balancer for Multitask Finetuning of Large Language Modelscited by 12
  11. 2023A Comprehensive Analysis of Adapter Efficiencycited by 8
  12. 2024Pillars of Grammatical Error Correction: Comprehensive Inspection Of Contemporary Approaches In The Era of Large Language Modelscited by 8
30 more
  1. 2024RVISA: Reasoning and Verification for Implicit Sentiment Analysiscited by 8
  2. 2022A Unified Encoder-Decoder Framework with Entity Memorycited by 7
  3. 2025LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Controlcited by 7
  4. 2023Pre-training Language Models for Comparative Reasoningcited by 6
  5. 2023When Multi-Task Learning Meets Partial Supervision: A Computer Vision Reviewcited by 6
  6. 2024Exploring Italian sentence embeddings properties through multi-taskingcited by 6
  7. 2024Task Diversity Shortens the ICL Plateaucited by 5
  8. 2024Identifying Task Groupings for Multi-Task Learning Using Pointwise V-Usable Informationcited by 4
  9. 2023STG-MTL: Scalable Task Grouping for Multi-Task Learning Using Data Mapcited by 3
  10. 2024Towards Principled Task Grouping for Multi-Task Learningcited by 3
  11. 2026STRUCTSURVEY: Structured Agentic Retrieval for Automated Survey Paper Generationcited by 3
  12. 2023Ask To The Point: Open-Domain Entity-Centric Question Generationcited by 2
  13. 2024M3TCM: Multi-modal Multi-task Context Model for Utterance Classification in Motivational Interviewscited by 2
  14. 2024How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classescited by 2
  15. 2025MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Modelscited by 2
  16. 2023Challenges and Opportunities of Using Transformer-Based Multi-Task Learning in NLP Through ML Lifecycle: A Surveycited by 1
  17. 2023FonMTL: Towards Multitask Learning for the Fon Languagecited by 1
  18. 2023Improving Cross-Domain Hate Speech Generalizability with Emotion Knowledgecited by 1
  19. 2024Stealthy Multi-Task Adversarial Attackscited by 1
  20. 2025HBO: Hierarchical Balancing Optimization for Fine-Tuning Large Language Modelscited by 1
  21. 2025Measuring Fine-Grained Relatedness in Multitask Learning via Data Attributioncited by 1
  22. 2025SAMO: A Lightweight Sharpness-Aware Approach for Multi-Task Optimization with Joint Global-Local Perturbationcited by 1
  23. 2026One LLM to Train Them All: Multi-Task Learning Framework for Fact-Checkingcited by 1
  24. 2026Aladdin-FTI @ AMIYA Three Wishes for Arabic NLP: Fidelity, Diglossia, and Multidialectal Generationcited by 1
  25. 2024Unplug and Play Language Models: Decomposing Experts in Language Models at Inference Timecited by 0
  26. 2025Efficient Inter-Task Attention for Multitask Transformer Modelscited by 0
  27. 2025Agentic Meta-Orchestrator for Multi-task Copilotscited by 0
  28. 2025Translation via Annotation: A Computational Study of Translating Classical Chinese into Japanesecited by 0
  29. 2026A Unified Model for Cross-Domain Clone Detection via Model Mergingcited by 0
  30. 2026Multi-Task Learning by using Contextualized Word Representations for Syntactic Parsing of a Morphologically Rich Languagecited by 0