# A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

arXiv: https://arxiv.org/abs/1611.01587 · HTML: https://mlsanity.com/paper/1611.01587/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2016.

## Built on by (most cited first, 30 of 62)

- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Multi-Task Learning as a Bargaining Game](https://mlsanity.com/paper/2202.01017/) (2022)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design](https://mlsanity.com/paper/2210.14793/) (2022)
- [Multi-Task Learning in Natural Language Processing: An Overview](https://mlsanity.com/paper/2109.09138/) (2021)
- [Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs](https://mlsanity.com/paper/2206.04674/) (2022)
- [The Next Chapter: A Study of Large Language Models in Storytelling](https://mlsanity.com/paper/2301.09790/) (2023)
- [BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning](https://mlsanity.com/paper/2206.08657/) (2022)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learning](https://mlsanity.com/paper/2302.11289/) (2023)
- [From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding](https://mlsanity.com/paper/2105.07316/) (2021)
- [A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods](https://mlsanity.com/paper/2204.03508/) (2022)
- [Annotating Columns with Pre-trained Language Models](https://mlsanity.com/paper/2104.01785/) (2021)
- [Three-Way Trade-Off in Multi-Objective Learning: Optimization, Generalization and Conflict-Avoidance](https://mlsanity.com/paper/2305.20057/) (2023)
- [Direction-oriented Multi-objective Learning: Simple and Provable Stochastic Algorithms](https://mlsanity.com/paper/2305.18409/) (2023)
- [Deep Multi-Task Learning for Joint Localization, Perception, and Prediction](https://mlsanity.com/paper/2101.06720/) (2021)
- [Specializing Multilingual Language Models: An Empirical Study](https://mlsanity.com/paper/2106.09063/) (2021)
- [Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases](https://mlsanity.com/paper/2112.07868/) (2021)
- [Collaborative Intelligence: Challenges and Opportunities](https://mlsanity.com/paper/2102.06841/) (2021)
- [Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text](https://mlsanity.com/paper/2112.07074/) (2021)
- [Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Stochastic Approach](https://mlsanity.com/paper/2210.12624/) (2022)
- [Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle](https://mlsanity.com/paper/2303.01980/) (2023)
- [PhoNLP: A joint multi-task learning model for Vietnamese part-of-speech tagging, named entity recognition and dependency parsing](https://mlsanity.com/paper/2101.01476/) (2021)
- [Meta-Learning for Effective Multi-task and Multilingual Modelling](https://mlsanity.com/paper/2101.10368/) (2021)
- [Paraphrase Identification with Deep Learning: A Review of Datasets and Methods](https://mlsanity.com/paper/2212.06933/) (2022)
- [Supervised Fine-tuning in turn Improves Visual Foundation Models](https://mlsanity.com/paper/2401.10222/) (2024)
- [Linguistically-Enriched and Context-Aware Zero-shot Slot Filling](https://mlsanity.com/paper/2101.06514/) (2021)
- [Auxiliary Learning as an Asymmetric Bargaining Game](https://mlsanity.com/paper/2301.13501/) (2023)
- [Same Same, But Different: Conditional Multi-Task Learning for Demographic-Specific Toxicity Detection](https://mlsanity.com/paper/2302.07372/) (2023)

## Abstract (arXiv)

Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks. Ideally, the linguistic levels of morphology, syntax and semantics would benefit each other by being trained in a single model. We introduce a joint many-task model together with a strategy for successively growing its depth to solve increasingly complex tasks. Higher layers include shortcut connections to lower-level task predictions to reflect linguistic hierarchies. We use a simple regularization term to allow for optimizing all model weights to improve one task's loss without exhibiting catastrophic interference of the other tasks. Our single end-to-end model obtains state-of-the-art or competitive results on five different tasks from tagging, parsing, relatedness, and entailment tasks.