mlsanityData to 7 Oct 2026

Multi-Task Learning in Natural Language Processing: An Overview

arXiv 2109.09138 ↗First posted 19 Sep 2021cs.AIcited by 61

Not on the map. Why?

Abstract (arXiv)

Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, directly training deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on these tasks, has been used to handle these problems. In this paper, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, gradient regularization, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 88 of its 173 references that are in the map. First posted on arXiv 19 Sep 2021.

Builds on

88 of 173 references.

References · 88

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  12. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
38 more
  1. 2017A Survey on Multi-Task Learningcited by 838
  2. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  3. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  4. 2017Dynamic Routing Between Capsulescited by 689
  5. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  6. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  7. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  8. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  9. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 221
  10. 2020Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Modelscited by 167
  11. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  12. 2015Multi-task Sequence to Sequence Learningcited by 94
  13. 2017Adversarial Multi-task Learning for Text Classificationcited by 91
  14. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  15. 2016A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Taskscited by 62
  16. 2018All-in-one: Multi-task Learning for Rumour Verificationcited by 54
  17. 2018A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Taskscited by 54
  18. 2018Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learningcited by 52
  19. 2019An Interactive Multi-Task Learning Network for End-to-End Aspect-Based Sentiment Analysiscited by 52
  20. 2020Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Datacited by 50
  21. 2019CopyMTL: Copy Mechanism for Joint Extraction of Entities and Relations with Multi-Task Learningcited by 47
  22. 2020Multi-task Learning for Multilingual Neural Machine Translationcited by 46
  23. 2016Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networkscited by 44
  24. 2015Natural Language Inference by Tree-Based Convolution and Heuristic Matchingcited by 42
  25. 2019Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extractioncited by 39
  26. 2014A Unified Perspective on Multi-Domain and Multi-Task Learningcited by 33
  27. 2019Multi-Task Learning for Conversational Question Answering over a Large-Scale Knowledge Basecited by 33
  28. 2018Multi-Task Neural Models for Translating Between Styles Within and Across Languagescited by 28
  29. 2018Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generationcited by 25
  30. 2018Dynamic Multi-Level Multi-Task Learning for Sentence Simplificationcited by 25
  31. 2019Continual and Multi-Task Architecture Searchcited by 22
  32. 2016When is multitask learning effective? Semantic sequence prediction under varying data conditionscited by 21
  33. 2017Semi-supervised Multitask Learning for Sequence Labelingcited by 20
  34. 2018A Neural Multi-Task Learning Framework to Jointly Model Medical Named Entity Recognition and Normalizationcited by 20
  35. 2019Multi-Task Self-Supervised Learning for Disfluency Detectioncited by 20
  36. 2019MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehensioncited by 20
  37. 2018Meta Multi-Task Learning for Sequence Modelingcited by 18
  38. 2019Neural Simile Recognition with Cyclic Multitask Learning and Local Attentioncited by 18

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 4; 2023: 6; 2024: 18; 2025: 25; 2026: 7.

61 papers.

Sort:

By area:

  1. 2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 65
  2. 2024Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challengescited by 42
  3. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 29
  4. 2023Dual-Balancing for Multi-Task Learningcited by 27
  5. 2022A Domain-adaptive Pre-training Approach for Language Bias Detection in Newscited by 16
  6. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 15
  7. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
  8. 2024FREE-Merging: Fourier Transform for Efficient Model Mergingcited by 12
  9. 2024GO4Align: Group Optimization for Multi-Task Alignmentcited by 11
  10. 2025MTL-UE: Learning to Learn Nothing for Multi-Task Learningcited by 11
  11. 2022Multi-Task Distributed Learning using Vision Transformer with Random Patch Permutationcited by 10
  12. 2023Precursor-of-Anomaly Detection for Irregular Time Seriescited by 9
38 more
  1. 2023UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructionscited by 9
  2. 2024Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasonercited by 9
  3. 2025MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learningcited by 9
  4. 2021Sharing to learn and learning to share; Fitting together Meta-Learning, Multi-Task Learning, and Transfer Learning: A meta reviewcited by 6
  5. 2023When Multi-Task Learning Meets Partial Supervision: A Computer Vision Reviewcited by 6
  6. 2024Exploring Italian sentence embeddings properties through multi-taskingcited by 6
  7. 2024Towards Better Multi-task Learning: A Framework for Optimizing Dataset Combinations in Large Language Modelscited by 6
  8. 2026TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddingscited by 6
  9. 2023Hate Speech and Offensive Language Detection using an Emotion-aware Shared Encodercited by 5
  10. 2024Explaining Veracity Predictions with Evidence Summarization: A Multi-Task Model Approachcited by 4
  11. 2024MAGPIE: Multi-Task Media-Bias Analysis Generalization for Pre-Trained Identification of Expressionscited by 4
  12. 2025Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacycited by 3
  13. 2025Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleavingcited by 3
  14. 2022Effective Cross-Task Transfer Learning for Explainable Natural Language Inference with T5cited by 2
  15. 2024SA-MDKIF: A Scalable and Adaptable Medical Domain Knowledge Injection Framework for Large Language Modelscited by 2
  16. 2025MaZO: Masked Zeroth-Order Optimization for Multi-Task Fine-Tuning of Large Language Modelscited by 2
  17. 2025Hexcute: A Compiler Framework for Automating Layout Synthesis in GPU Programscited by 2
  18. 2025Multi-Domain Explainability of Preferencescited by 2
  19. 2025Fairness Metric Design Exploration in Multi-Domain Moral Sentiment Classification using Transformer-Based Modelscited by 2
  20. 2025NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspectivecited by 2
  21. 2026UV-M3TL: A Unified and Versatile Multimodal Multi-Task Learning Framework for Assistive Driving Perceptioncited by 2
  22. 2023Challenges and Opportunities of Using Transformer-Based Multi-Task Learning in NLP Through ML Lifecycle: A Surveycited by 1
  23. 2024Deep multitask neural networks for solving some stochastic optimal control problemscited by 1
  24. 2025PhenoProfiler: Advancing Phenotypic Learning for Image-based Drug Discoverycited by 1
  25. 2025CLaC at SemEval-2025 Task 6: A Multi-Architecture Approach for Corporate Environmental Promise Verificationcited by 1
  26. 2025Towards Unified Modeling in Federated Multi-Task Learning via Subspace Decouplingcited by 1
  27. 2025SAMO: A Lightweight Sharpness-Aware Approach for Multi-Task Optimization with Joint Global-Local Perturbationcited by 1
  28. 2025Small transformer architectures for task switchingcited by 1
  29. 2025POLIS-Bench: Towards Multi-Dimensional Evaluation of LLMs for Bilingual Policy Tasks in Governmental Scenarioscited by 1
  30. 2026Dynamic Model Merging Made Slimcited by 1
  31. 2024Unplug and Play Language Models: Decomposing Experts in Language Models at Inference Timecited by 0
  32. 2024MTLComb: multi-task learning combining regression and classification tasks for joint feature selectioncited by 0
  33. 2024Intertwined Biases Across Social Media Spheres: Unpacking Correlations in Media Bias Dimensionscited by 0
  34. 2024Building a Multivariate Time Series Benchmarking Datasets Inspired by Natural Language Processing (NLP)cited by 0
  35. 2024E2E-AFG: An End-to-End Model with Adaptive Filtering for Retrieval-Augmented Generationcited by 0
  36. 2024Extreme Multi-label Completion for Semantic Document Labelling with Taxonomy-Aware Parallel Learningcited by 0
  37. 2025Sharper Risk Bound for Multi-Task Learning with Multi-Graph Dependent Datacited by 0
  38. 2025Multi-Level Collaboration in Model Mergingcited by 0

50 most cited of 61 All 61 (JSON)