mlsanityData to 14 Oct 2026

Speech and audio distillation

197 papers1 this weekHigh confidence in this grouping: the papers form one clear topicHigh confidence in this grouping: the papers form one clear topic · About

Keywords: speech, distillation, audio, knowledge, asr

Key facts. Speech and audio distillation: 197 papers on arXiv since 2021 (data through 14 Oct 2026). 14 of its 197 mapped papers (7%) cite “Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling” (2023, arXiv 2311.00430). 5 new papers 25 Sep–14 Oct.

What’s new

25 Sep – 14 Oct 2026

5 new papers, 25 Sep – 14 Oct; 1 this week.

  1. This week · 1
  2. CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognition
  3. Earlier · 4
  4. Learning from imperfect teachers for low-resource acoustic generalization also fits: General knowledge distillation methods
  5. DriftTTS: Few-Step Text-to-Speech Without Distillation via Distribution-Matching Drift
  6. OP-CAD: On-Policy Clean-Audio Distillation for Robust Audio-Visual Reasoning also fits: Multimodal distillation
  7. Distill Locally, Schedule Globally: Flow Maps for Few-Step Text-to-Speech

Foundations

Most cited by papers here

  1. Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling2023cited by 14 here
  2. Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models2021cited by 10 here
  3. Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models2022cited by 8 here
  4. CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition2021cited by 6 here
  5. Two-Step Knowledge Distillation for Tiny Speech Enhancement2023cited by 6 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2021CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognitioncited by 6 here
  2. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Modelscited by 10 here
  3. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 8 here
  4. 2023Ensemble knowledge distillation of self-supervised speech modelscited by 5 here
20 more it builds on, not in this chain
  1. 2021Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognitioncited by 4 here
  2. 2021Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translationcited by 3 here
  3. 2021Distilling Knowledge via Knowledge Reviewcited by 3 here
  4. 2021Knowledge distillation: A good teacher is patient and consistentcited by 6 here
  5. 2021Temporal Knowledge Distillation for On-device Audio Classificationcited by 4 here
  6. 2022Two-Pass End-to-End ASR Model Compressioncited by 3 here
  7. 2022CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classificationcited by 3 here
  8. 2022Decoupled Knowledge Distillationcited by 6 here
  9. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 3 here
  10. 2022Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillationcited by 5 here
  11. 2022Conformer with dual-mode chunked attention for joint online and offline ASRcited by 3 here
  12. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 5 here
  13. 2022Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognitioncited by 5 here
  14. 2022Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillationcited by 4 here
  15. 2022Inter-KD: Intermediate Knowledge Distillation for CTC-Based Automatic Speech Recognitioncited by 4 here
  16. 2023Learning From Yourself: A Self-Distillation Method for Fake Speech Detectioncited by 3 here
  17. 2023ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppressioncited by 4 here
  18. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 5 here
  19. 2023Two-Step Knowledge Distillation for Tiny Speech Enhancementcited by 6 here
  20. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 14 here

The 24 most cited of 153.

Browse

30 of 197, ranked 6–35

  1. 6Ensemble knowledge distillation of self-supervised speech models20235
  2. 7Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillation20225
  3. 8FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning20225
  4. 9Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition20225
  5. 10ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression20234
  6. 11Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation20224
  7. 12Inter-KD: Intermediate Knowledge Distillation for CTC-Based Automatic Speech Recognition20224
  8. 13Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition20214
  9. 14Temporal Knowledge Distillation for On-device Audio Classification20214
  10. 15Learning From Yourself: A Self-Distillation Method for Fake Speech Detection20233
All 30
  1. 16Two-Pass End-to-End ASR Model Compression20223
  2. 17CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification20223
  3. 18Conformer with dual-mode chunked attention for joint online and offline ASR20223
  4. 19Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation20213
  5. 20X3-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment20262
  6. 21Dynamic Frequency-Adaptive Knowledge Distillation for Speech Enhancement20252
  7. 22Distil-DCCRN: A Small-footprint DCCRN Leveraging Feature-based Knowledge Distillation in Speech Enhancement20242
  8. 23Towards domain generalisation in ASR with elitist sampling and ensemble knowledge distillation20232
  9. 24One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification20232
  10. 25Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer20232
  11. 26Leveraging ASR Pretrained Conformers for Speaker Verification through Transfer Learning and Knowledge Distillation20232
  12. 27Emphasized Non-Target Speaker Knowledge in Knowledge Distillation for Automatic Speaker Verification20232
  13. 28Dual-Branch Knowledge Distillation for Noise-Robust Synthetic Speech Detection20232
  14. 29Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR20222
  15. 30Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation20222
  16. 31Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillation20222
  17. 32Test-Time Adaptation Toward Personalized Speech Enhancement: Zero-Shot Learning with Knowledge Distillation20212
  18. 33Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classification20212
  19. 34Joint Multiple Intent Detection and Slot Filling via Self-distillation20212
  20. 35ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition20261