# Speech and audio distillation

Area: [Knowledge distillation](https://mlsanity.com/distill/index.md). HTML: https://mlsanity.com/distill/speech-and-audio-distillation/

## Key facts

- Speech and audio distillation: 197 papers on arXiv since 2021 (data through 14 Oct 2026).
- 14 of its 197 mapped papers (7%) cite “Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling” (2023, arXiv 2311.00430).
- 10 of its 197 mapped papers (5%) cite “Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models” (2021, arXiv 2110.03334).
- 5 new papers 25 Sep–14 Oct.

## Most cited here

- [Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://mlsanity.com/paper/2311.00430/) (2023), cited by 14 papers here
- [Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models](https://mlsanity.com/paper/2110.03334/) (2021), cited by 10 papers here
- [Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models](https://mlsanity.com/paper/2207.06867/) (2022), cited by 8 papers here
- [CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition](https://mlsanity.com/paper/2106.07734/) (2021), cited by 6 papers here
- [Two-Step Knowledge Distillation for Tiny Speech Enhancement](https://mlsanity.com/paper/2309.08144/) (2023), cited by 6 papers here
- [Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillation](https://mlsanity.com/paper/2204.00771/) (2022), cited by 5 papers here
- [FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning](https://mlsanity.com/paper/2207.00555/) (2022), cited by 5 papers here
- [Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition](https://mlsanity.com/paper/2210.00169/) (2022), cited by 5 papers here
- [Ensemble knowledge distillation of self-supervised speech models](https://mlsanity.com/paper/2302.12757/) (2023), cited by 5 papers here
- [Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition](https://mlsanity.com/paper/2103.00422/) (2021), cited by 4 papers here

## New (25 Sep–14 Oct)

- [CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition](https://mlsanity.com/paper/2610.19134/) (2026-10-13)
- [Learning from imperfect teachers for low-resource acoustic generalization](https://mlsanity.com/paper/2610.05256/) (2026-10-04)
- [DriftTTS: Few-Step Text-to-Speech Without Distillation via Distribution-Matching Drift](https://mlsanity.com/paper/2610.03390/) (2026-10-02)
- [OP-CAD: On-Policy Clean-Audio Distillation for Robust Audio-Visual Reasoning](https://mlsanity.com/paper/2609.39150/) (2026-09-30)
- [Distill Locally, Schedule Globally: Flow Maps for Few-Step Text-to-Speech](https://mlsanity.com/paper/2609.36324/) (2026-09-28)