# Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding

arXiv: https://arxiv.org/abs/1904.09482 · HTML: https://mlsanity.com/paper/1904.09482/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2019.

## Built on by (most cited first, 30 of 37)

- [Learning by Distilling Context](https://mlsanity.com/paper/2209.15189/) (2022)
- [Multi-Task Self-Training for Learning General Representations](https://mlsanity.com/paper/2108.11353/) (2021)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks](https://mlsanity.com/paper/2206.06561/) (2022)
- [Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text Summarization](https://mlsanity.com/paper/2112.11670/) (2021)
- [Survey on Automated Short Answer Grading with Deep Learning: from Word Embeddings to Transformers](https://mlsanity.com/paper/2204.03503/) (2022)
- [XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation](https://mlsanity.com/paper/2106.04563/) (2021)
- [Explaining Bayesian Neural Networks](https://mlsanity.com/paper/2108.10346/) (2021)
- [When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning](https://mlsanity.com/paper/2205.08124/) (2022)
- [Protein Language Models and Structure Prediction: Connection and Progression](https://mlsanity.com/paper/2211.16742/) (2022)
- [Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection](https://mlsanity.com/paper/2311.01755/) (2023)
- [Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data](https://mlsanity.com/paper/2410.18588/) (2024)
- [FLamE: Few-shot Learning from Natural Language Explanations](https://mlsanity.com/paper/2306.08042/) (2023)
- [Dual-View Distilled BERT for Sentence Embedding](https://mlsanity.com/paper/2104.08675/) (2021)
- [Improving Question Answering Performance Using Knowledge Distillation and Active Learning](https://mlsanity.com/paper/2109.12662/) (2021)
- [Graph-based Knowledge Distillation: A survey and experimental evaluation](https://mlsanity.com/paper/2302.14643/) (2023)
- [Unified and Effective Ensemble Knowledge Distillation](https://mlsanity.com/paper/2204.00548/) (2022)
- [Multi-Objective Optimization for Sparse Deep Multi-Task Learning](https://mlsanity.com/paper/2308.12243/) (2023)
- [The Impact of Artificial Intelligence on the Evolution of Digital Education: A Comparative Study of OpenAI Text Generation Tools including ChatGPT, Bing Chat, Bard, and Ernie](https://mlsanity.com/paper/2309.02029/) (2023)
- [Industry Scale Semi-Supervised Learning for Natural Language Understanding](https://mlsanity.com/paper/2103.15871/) (2021)
- [HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression](https://mlsanity.com/paper/2110.08551/) (2021)
- [BERT-DRE: BERT with Deep Recursive Encoder for Natural Language Sentence Matching](https://mlsanity.com/paper/2111.02188/) (2021)
- [Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention](https://mlsanity.com/paper/2204.03479/) (2022)
- [SPICED: News Similarity Detection Dataset with Multiple Topics and Complexity Levels](https://mlsanity.com/paper/2309.13080/) (2023)
- [Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models](https://mlsanity.com/paper/2310.13315/) (2023)
- [Measuring Social Biases in Masked Language Models by Proxy of Prediction Quality](https://mlsanity.com/paper/2402.13954/) (2024)
- [Distill-then-prune: An Efficient Compression Framework for Real-time Stereo Matching Network on Edge Devices](https://mlsanity.com/paper/2405.11809/) (2024)
- [Leveraging Sentiment Analysis Knowledge to Solve Emotion Detection Tasks](https://mlsanity.com/paper/2111.03715/) (2021)
- [A Comparison of Document Similarity Algorithms](https://mlsanity.com/paper/2304.01330/) (2023)
- [minOffense: Inter-Agreement Hate Terms for Stable Rules, Concepts, Transitivities, and Lattices](https://mlsanity.com/paper/2305.17984/) (2023)

## Abstract (arXiv)

This paper explores the use of knowledge distillation to improve a Multi-Task Deep Neural Network (MT-DNN) (Liu et al., 2019) for learning text representations across multiple natural language understanding tasks. Although ensemble learning can improve model performance, serving an ensemble of large DNNs such as MT-DNN can be prohibitively expensive. Here we apply the knowledge distillation method (Hinton et al., 2015) in the multi-task learning setting. For each task, we train an ensemble of different MT-DNNs (teacher) that outperforms any single model, and then train a single MT-DNN (student) via multi-task learning to distill knowledge from these ensemble teachers. We show that the distilled MT-DNN significantly outperforms the original MT-DNN on 7 out of 9 GLUE tasks, pushing the GLUE benchmark (single model) to 83.7% (1.5% absolute improvementfootnote Based on the GLUE leaderboard at https://gluebenchmark.com/leaderboard as of April 1, 2019.). The code and pre-trained models will be made publicly available at https://github.com/namisan/mt-dnn.