# MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models

arXiv: https://arxiv.org/abs/1911.03588 · HTML: https://mlsanity.com/paper/1911.03588/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2019.

## Built on by (most cited first, 8 of 8)

- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models](https://mlsanity.com/paper/2203.12758/) (2022)
- [MergeDistill: Merging Pre-trained Language Models using Distillation](https://mlsanity.com/paper/2106.02834/) (2021)
- [Differentially Private Model Compression](https://mlsanity.com/paper/2206.01838/) (2022)
- [HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression](https://mlsanity.com/paper/2110.08551/) (2021)
- [ERNIE 3.0 Tiny: Frustratingly Simple Method to Improve Task-Agnostic Distillation Generalization](https://mlsanity.com/paper/2301.03416/) (2023)
- [Efficient Sparse Selective-Update RNNs for Long-Range Sequence Modeling](https://mlsanity.com/paper/2603.02226/) (2026)
- [Empirical Boundaries of Reservoir Computing for Few-Shot Sequence Pattern Tasks](https://mlsanity.com/paper/2610.19997/) (2026)

## Abstract (arXiv)

Pretrained language models have led to significant performance gains in many NLP tasks. However, the intensive computing resources to train such models remain an issue. Knowledge distillation alleviates this problem by learning a light-weight student model. So far the distillation approaches are all task-specific. In this paper, we explore knowledge distillation under the multi-task learning setting. The student is jointly distilled across different tasks. It acquires more general representation capacity through multi-tasking distillation and can be further fine-tuned to improve the model in the target domain. Unlike other BERT distillation methods which specifically designed for Transformer-based architectures, we provide a general learning framework. Our approach is model agnostic and can be easily applied on different future teacher model architectures. We evaluate our approach on a Transformer-based and LSTM based student model. Compared to a strong, similarly LSTM-based approach, we achieve better quality under the same computational constraints. Compared to the present state of the art, we reach comparable results with much faster inference speed.