# Multi-Task Deep Neural Networks for Natural Language Understanding

arXiv: https://arxiv.org/abs/1901.11504 · HTML: https://mlsanity.com/paper/1901.11504/

## Key facts

- Cited by 350 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2019.

## Built on by (most cited first, 30 of 350)

- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks](https://mlsanity.com/paper/2206.08916/) (2022)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [AdaMerging: Adaptive Model Merging for Multi-Task Learning](https://mlsanity.com/paper/2310.02575/) (2023)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [Muppet: Massive Multi-task Representations with Pre-Finetuning](https://mlsanity.com/paper/2101.11038/) (2021)
- [Meta-learning via Language Model In-context Tuning](https://mlsanity.com/paper/2110.07814/) (2021)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)
- [Task-Specific Skill Localization in Fine-tuned Language Models](https://mlsanity.com/paper/2302.06600/) (2023)
- [Graph Foundation Models: Concepts, Opportunities and Challenges](https://mlsanity.com/paper/2310.11829/) (2023)
- [UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark](https://mlsanity.com/paper/2103.13009/) (2021)
- [M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design](https://mlsanity.com/paper/2210.14793/) (2022)
- [Language Models are General-Purpose Interfaces](https://mlsanity.com/paper/2206.06336/) (2022)
- [Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases](https://mlsanity.com/paper/2303.14742/) (2023)
- [ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts](https://mlsanity.com/paper/2205.11961/) (2022)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation](https://mlsanity.com/paper/2109.05729/) (2021)
- [Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPT](https://mlsanity.com/paper/2304.11116/) (2023)
- [PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance](https://mlsanity.com/paper/2206.12562/) (2022)
- [A Survey on Stance Detection for Mis- and Disinformation Identification](https://mlsanity.com/paper/2103.00242/) (2021)
- [Independent Component Alignment for Multi-Task Learning](https://mlsanity.com/paper/2305.19000/) (2023)

## Abstract (arXiv)

In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks. MT-DNN not only leverages large amounts of cross-task data, but also benefits from a regularization effect that leads to more general representations in order to adapt to new tasks and domains. MT-DNN extends the model proposed in Liu et al. (2015) by incorporating a pre-trained bidirectional transformer language model, known as BERT (Devlin et al., 2018). MT-DNN obtains new state-of-the-art results on ten NLU tasks, including SNLI, SciTail, and eight out of nine GLUE tasks, pushing the GLUE benchmark to 82.7% (2.2% absolute improvement). We also demonstrate using the SNLI and SciTail datasets that the representations learned by MT-DNN allow domain adaptation with substantially fewer in-domain labels than the pre-trained BERT representations. The code and pre-trained models are publicly available at https://github.com/namisan/mt-dnn.