# DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognition

arXiv: https://arxiv.org/abs/2305.10788 · HTML: https://mlsanity.com/paper/2305.10788/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 18 May 2023.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Structured Pruning Learns Compact and Accurate Models](https://mlsanity.com/paper/2204.00408/) (2022)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT](https://mlsanity.com/paper/2203.15610/) (2022)
- [Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding](https://mlsanity.com/paper/2302.14132/) (2023)
- [The History of Speech Recognition to the Year 2030](https://mlsanity.com/paper/2108.00084/) (2021)

## Built on by (most cited first, 11 of 11)

- [Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://mlsanity.com/paper/2311.00430/) (2023)
- [Whisper in Medusa's Ear: Multi-head Efficient Decoding for Transformer-based ASR](https://mlsanity.com/paper/2409.15869/) (2024)
- [Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languages](https://mlsanity.com/paper/2503.23542/) (2025)
- [To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation](https://mlsanity.com/paper/2406.04512/) (2024)
- [AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weighting](https://mlsanity.com/paper/2405.08019/) (2024)
- [Outlier Reduction with Gated Attention for Improved Post-training Quantization in Large Sequence-to-sequence Speech Foundation Models](https://mlsanity.com/paper/2406.11022/) (2024)
- [QUADS: QUAntized Distillation Framework for Efficient Speech Language Understanding](https://mlsanity.com/paper/2505.14723/) (2025)
- [AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformers](https://mlsanity.com/paper/2406.08904/) (2024)
- [UniBERT: Adversarial Training for Language-Universal Representations](https://mlsanity.com/paper/2503.12608/) (2025)
- [Novel Parasitic Dual-Scale Modeling for Efficient and Accurate Multilingual Speech Translation](https://mlsanity.com/paper/2508.11189/) (2025)
- [MURMUR: An Efficient Inference System for Long-Form ASR](https://mlsanity.com/paper/2606.01483/) (2026)

## Abstract (arXiv)

As a popular multilingual and multitask pre-trained speech model, Whisper has the problem of curse of multilinguality. To enhance multilingual capabilities in small Whisper models, we propose DQ-Whisper, a novel joint distillation and quantization framework to compress Whisper for efficient inference. Firstly, we propose a novel dynamic matching distillation strategy. Then, a quantization-aware distillation framework is introduced to integrate quantization with distillation. Experimental results on various multilingual datasets show that our suggested distillation approach can effectively enhance the multilingual capabilities of small Whisper models without increasing computational costs. Up to 5.18x reduction in model size is achieved with marginal performance degradation. In addition, quantization is compatible with distillation, which can result in a higher compression rate.