# Multi-task learning for natural language processing in the 2020s: where are we going?

arXiv: https://arxiv.org/abs/2007.16008 · HTML: https://mlsanity.com/paper/2007.16008/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2020.

## Built on by (most cited first, 27 of 27)

- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [MEDIC: A Multi-Task Learning Dataset for Disaster Image Classification](https://mlsanity.com/paper/2108.12828/) (2021)
- [A Domain-adaptive Pre-training Approach for Language Bias Detection in News](https://mlsanity.com/paper/2205.10773/) (2022)
- [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://mlsanity.com/paper/2206.12131/) (2022)
- [Less is More: Selective Layer Finetuning with SubTuning](https://mlsanity.com/paper/2302.06354/) (2023)
- [Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processing](https://mlsanity.com/paper/2105.14839/) (2021)
- [A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trends](https://mlsanity.com/paper/2302.03512/) (2023)
- [Learning to Transfer for Traffic Forecasting via Multi-task Learning](https://mlsanity.com/paper/2111.15542/) (2021)
- [WASSA@IITK at WASSA 2021: Multi-task Learning and Transformer Finetuning for Emotion Classification and Empathy Prediction](https://mlsanity.com/paper/2104.09827/) (2021)
- [Learning to Bridge Metric Spaces: Few-shot Joint Learning of Intent Detection and Slot Filling](https://mlsanity.com/paper/2106.07343/) (2021)
- [Transfer Learning Study of Motion Transformer-based Trajectory Predictions](https://mlsanity.com/paper/2404.08271/) (2024)
- [Generative multitask learning mitigates target-causing confounding](https://mlsanity.com/paper/2202.04136/) (2022)
- [OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue System](https://mlsanity.com/paper/2312.16864/) (2023)
- [Bench-Marking And Improving Arabic Automatic Image Captioning Through The Use Of Multi-Task Learning Paradigm](https://mlsanity.com/paper/2202.05474/) (2022)
- [Cold Start Streaming Learning for Deep Networks](https://mlsanity.com/paper/2211.04624/) (2022)
- [Efficient Extraction of Pathologies from C-Spine Radiology Reports using Multi-Task Learning](https://mlsanity.com/paper/2204.04544/) (2022)
- [Explaining the Effectiveness of Multi-Task Learning for Efficient Knowledge Extraction from Spine MRI Reports](https://mlsanity.com/paper/2205.02979/) (2022)
- [Challenges and Opportunities of Using Transformer-Based Multi-Task Learning in NLP Through ML Lifecycle: A Survey](https://mlsanity.com/paper/2308.08234/) (2023)
- [A Unified Causal View of Instruction Tuning](https://mlsanity.com/paper/2402.06220/) (2024)
- [Multi-Task Learning for Features Extraction in Financial Annual Reports](https://mlsanity.com/paper/2404.05281/) (2024)
- [Measuring Fine-Grained Relatedness in Multitask Learning via Data Attribution](https://mlsanity.com/paper/2505.21438/) (2025)
- [New Tight Relaxations of Rank Minimization for Multi-Task Learning](https://mlsanity.com/paper/2112.04734/) (2021)
- [TCBERT: A Technical Report for Chinese Topic Classification BERT](https://mlsanity.com/paper/2211.11304/) (2022)
- [Extending Memory for Language Modelling](https://mlsanity.com/paper/2305.11462/) (2023)
- [Mitigating Negative Transfer with Task Awareness for Sexism, Hate Speech, and Toxic Language Detection](https://mlsanity.com/paper/2307.03377/) (2023)
- [SUM: Unified Geometric Surgery on Spatio-Temporal Adaptation Vectors for Federated Class Incremental Learning](https://mlsanity.com/paper/2607.19384/) (2026)

## Abstract (arXiv)

Multi-task learning (MTL) significantly pre-dates the deep learning era, and it has seen a resurgence in the past few years as researchers have been applying MTL to deep learning solutions for natural language tasks. While steady MTL research has always been present, there is a growing interest driven by the impressive successes published in the related fields of transfer learning and pre-training, such as BERT, and the release of new challenge problems, such as GLUE and the NLP Decathlon (decaNLP). These efforts place more focus on how weights are shared across networks, evaluate the re-usability of network components and identify use cases where MTL can significantly outperform single-task solutions. This paper strives to provide a comprehensive survey of the numerous recent MTL contributions to the field of natural language processing and provide a forum to focus efforts on the hardest unsolved problems in the next decade. While novel models that improve performance on NLP benchmarks are continually produced, lasting MTL challenges remain unsolved which could hold the key to better language understanding, knowledge discovery and natural language interfaces.