Data to 7 Oct 2026

Multi-Task Deep Neural Networks for Natural Language Understanding

arXiv 1901.11504 ↗First posted 31 Jan 2019cs.CLcited by 349

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks. MT-DNN not only leverages large amounts of cross-task data, but also benefits from a regularization effect that leads to more general representations in order to adapt to new tasks and domains. MT-DNN extends the model proposed in Liu et al. (2015) by incorporating a pre-trained bidirectional transformer language model, known as BERT (Devlin et al., 2018). MT-DNN obtains new state-of-the-art results on ten NLU tasks, including SNLI, SciTail, and eight out of nine GLUE tasks, pushing the GLUE benchmark to 82.7% (2.2% absolute improvement). We also demonstrate using the SNLI and SciTail datasets that the representations learned by MT-DNN allow domain adaptation with substantially fewer in-domain labels than the pre-trained BERT representations. The code and pre-trained models are publicly available at https://github.com/namisan/mt-dnn.

Key facts. Cited by 349 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 109; 2022: 94; 2023: 74; 2024: 42; 2025: 23; 2026: 7.

349 papers.

Sort:

By area:

2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
2023A Survey of Large Language Modelscited by 2,537
2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
2021True Few-Shot Learning with Language Modelscited by 343
2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195

38 more

2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
2021Meta-learning via Language Model In-context Tuningcited by 163
2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
2022Language Models are General-Purpose Interfacescited by 96
2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 85
2023Modular Deep Learningcited by 82
2021CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generationcited by 81
2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
2023Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPTcited by 78
2021A Survey on Stance Detection for Mis- and Disinformation Identificationcited by 75
2023Independent Component Alignment for Multi-Task Learningcited by 69
2021FLEX: Unifying Evaluation for Few-Shot NLPcited by 67
2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
2021ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Informationcited by 65
2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
2022Transferability in Deep Learning: A Surveycited by 57
2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
2021Automated Fact-Checking: A Surveycited by 55
2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
2021Near-optimal Representation Learning for Linear Bandits and Linear RLcited by 45
2021Multi-task Retrieval for Knowledge-Intensive Taskscited by 44
2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
2023One-Shot Learning as Instruction Data Prospector for Large Language Modelscited by 43
2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
2021TreeBERT: A Tree-Based Pre-Trained Model for Programming Languagecited by 40
2021DeepXML: A Deep Extreme Multi-Label Learning Framework Applied to Short Text Documentscited by 37
2022KOBEST: Korean Balanced Evaluation of Significant Taskscited by 37
2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
2021Multilingual and Cross-Lingual Intent Detection from Spoken Datacited by 35

50 most cited of 349 All 349 (JSON)