# Model Compression with Multi-Task Knowledge Distillation for Web-scale Question Answering System

arXiv: https://arxiv.org/abs/1904.09636 · HTML: https://mlsanity.com/paper/1904.09636/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2019.

## Built on by (most cited first, 5 of 5)

- [AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning](https://mlsanity.com/paper/2102.01386/) (2021)
- [Improving Question Answering Performance Using Knowledge Distillation and Active Learning](https://mlsanity.com/paper/2109.12662/) (2021)
- [Larger models yield better results? Streamlined severity classification of ADHD-related concerns using BERT-based knowledge distillation](https://mlsanity.com/paper/2411.00052/) (2024)
- [General Cross-Architecture Distillation of Pretrained Language Models into Matrix Embeddings](https://mlsanity.com/paper/2109.08449/) (2021)
- [Compressing Cross-Lingual Multi-Task Models at Qualtrics](https://mlsanity.com/paper/2211.15927/) (2022)

## Abstract (arXiv)

Deep pre-training and fine-tuning models (like BERT, OpenAI GPT) have demonstrated excellent results in question answering areas. However, due to the sheer amount of model parameters, the inference speed of these models is very slow. How to apply these complex models to real business scenarios becomes a challenging but practical problem. Previous works often leverage model compression approaches to resolve this problem. However, these methods usually induce information loss during the model compression procedure, leading to incomparable results between compressed model and the original model. To tackle this challenge, we propose a Multi-task Knowledge Distillation Model (MKDM for short) for web-scale Question Answering system, by distilling knowledge from multiple teacher models to a light-weight student model. In this way, more generalized knowledge can be transferred. The experiment results show that our method can significantly outperform the baseline methods and even achieve comparable results with the original teacher models, along with significant speedup of model inference.