mlsanityData to 7 Oct 2026

Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling

arXiv 2311.00430 ↗First posted 1 Nov 2023cs.CL cs.SD eess.AScited by 87

Not on the map. Why?

Abstract (arXiv)

As the size of pre-trained speech recognition models increases, running these large models in low-latency or resource-constrained environments becomes challenging. In this work, we leverage pseudo-labelling to assemble a large-scale open-source dataset which we use to distill the Whisper model into a smaller variant, called Distil-Whisper. Using a simple word error rate (WER) heuristic, we select only the highest quality pseudo-labels for training. The distilled model is 5.8 times faster with 51% fewer parameters, while performing to within 1% WER on out-of-distribution test data in a zero-shot transfer setting. Distil-Whisper maintains the robustness of the Whisper model to difficult acoustic conditions, while being less prone to hallucination errors on long-form audio. Distil-Whisper is designed to be paired with Whisper for speculative decoding, yielding a 2 times speed-up while mathematically ensuring the same outputs as the original model. To facilitate further research in this domain, we make our training code, inference code and models publicly accessible.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Builds on 46 of its 73 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

46 of 73 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  12. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
34 more
  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  2. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  5. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  7. 2016Sequence-Level Knowledge Distillationcited by 761
  8. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  9. 2015Listen, Attend and Spellcited by 542
  10. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  11. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  12. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  13. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  14. 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
  15. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
  16. 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 320
  17. 2022Confident Adaptive Language Modelingcited by 297
  18. 2018Streaming End-to-end Speech Recognition For Mobile Devicescited by 230
  19. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  20. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  21. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
  22. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  23. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 98
  24. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  25. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  26. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 77
  27. 2020Pre-trained Summarization Distillationcited by 70
  28. 2021Earnings-21: A Practical Benchmark for ASR in the Wildcited by 60
  29. 2022Earnings-22: A Practical Benchmark for Accents in the Wildcited by 46
  30. 2022DQ-BART: Efficient Sequence-to-Sequence Model via Joint Distillation and Quantizationcited by 23
  31. 2022ESB: A Benchmark For Multi-Domain End-to-End Speech Recognitioncited by 18
  32. 2021Shrinking Bigfoot: Reducing wav2vec 2.0 footprintcited by 16
  33. 2021Robustness Challenges in Model Distillation and Pruning for Natural Language Understandingcited by 12
  34. 2023DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognitioncited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 35; 2026: 30.

87 papers.

Sort:

By area:

  1. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 76
  2. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  3. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 50
  4. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  5. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 24
  6. 2025Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluationcited by 21
  7. 2024Distillation of Diffusion Features for Semantic Correspondencecited by 17
  8. 2025Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Modelscited by 17
  9. 2025Scaling Rich Style-Prompted Text-to-Speech Datasetscited by 17
  10. 2024The BabyView dataset: High-resolution egocentric videos of infants' and young children's everyday experiencescited by 15
  11. 2025TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modelingcited by 15
  12. 2025Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Downcited by 12
38 more
  1. 2025Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyondcited by 10
  2. 2025Training and Inference Efficiency of Encoder-Decoder Speech Modelscited by 8
  3. 2024NeKo: Cross-Modality Post-Recognition Error Correction with Tasks-Guided Mixture-of-Experts Language Modelcited by 7
  4. 2025LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximationcited by 7
  5. 2024Whisper in Medusa's Ear: Multi-head Efficient Decoding for Transformer-based ASRcited by 6
  6. 2025SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understandingcited by 6
  7. 2024Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Datacited by 5
  8. 2024MPC-Minimized Secure LLM Inferencecited by 5
  9. 2024Whispering in Norwegian: Navigating Orthographic and Dialectic Challengescited by 4
  10. 2024To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillationcited by 4
  11. 2024The Greek podcast corpus: Competitive speech models for low-resourced languages with weakly supervised datacited by 4
  12. 2025Whisper-LM: Improving ASR Models with Language Models for Low-Resource Languagescited by 4
  13. 2025Efficient Speech Translation through Model Compression and Knowledge Distillationcited by 4
  14. 2025BaldWhisper: Faster Whisper with Head Shearing and Layer Mergingcited by 4
  15. 2025Iterative Layer Pruning for Efficient Translation Inferencecited by 4
  16. 2024uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimescited by 3
  17. 2024A Large Dataset of Spontaneous Speech with the Accent Spoken in S~ao Paulo for Automatic Speech Recognition Evaluationcited by 3
  18. 2024CLOVER: Cross-Layer Orthogonal Vectors Pruning and Fine-Tuningcited by 3
  19. 2024MathSpeech: Leveraging Small LMs for Accurate Conversion in Mathematical Speech-to-Formulacited by 3
  20. 2025Assessing the alignment between infants' visual and linguistic experience using multimodal language modelscited by 3
  21. 2024Optimized Speculative Sampling for GPU Hardware Acceleratorscited by 2
  22. 2024Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLMcited by 2
  23. 2024ELAICHI: Enhancing Low-resource TTS by Addressing Infrequent and Low-frequency Character Bigramscited by 2
  24. 2025On the Robust Approximation of ASR Metricscited by 2
  25. 2025Vedavani: A Benchmark Corpus for ASR on Vedic Sanskrit Poetrycited by 2
  26. 2025Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasiacited by 2
  27. 2025Early Attentive Sparsification Accelerates Neural Speech Transcriptioncited by 2
  28. 2025DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objectivecited by 2
  29. 2025Tiny is not small enough: High-quality, low-resource facial animation models through hybrid knowledge distillationcited by 2
  30. 2025Adaptive Knowledge Distillation for Device-Directed Speech Detectioncited by 2
  31. 2025CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generationcited by 2
  32. 2025Re-evaluating Minimum Bayes Risk Decoding for Automatic Speech Recognitioncited by 2
  33. 2026AfriNLLB: Efficient Translation Models for African Languagescited by 2
  34. 2026On the Role of Encoder Depth: Pruning Whisper and LoRA Fine-Tuning in SLAM-ASRcited by 2
  35. 2026Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Modelscited by 2
  36. 2024Deep Learning Models in Speech Recognition: Measuring GPU Energy Consumption, Impact of Noise and Model Quantization for Edge Deploymentcited by 1
  37. 2024Towards Effective and Efficient Non-autoregressive Decoding Using Block-based Attention Maskcited by 1
  38. 2024Fast Streaming Transducer ASR Prototyping via Knowledge Distillation with Whispercited by 1

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation
  2. 2026Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding ↗
  3. 2026FlowW2N: Whispered-to-Normal Speech Conversion via Flow-Matching
  4. 2023Teach me with a Whisper: Enhancing Large Language Models for Analyzing Spoken Transcripts using Speech Embeddings
  5. 2025DiffSoundStream: Efficient Speech Tokenization via Diffusion Decoding
  6. 2026DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding
  7. 2024Keep Decoding Parallel with Effective Knowledge Distillation from Language Models to End-to-end Speech Recognisers
  8. 2025dParallel: Learnable Parallel Decoding for dLLMs