mlsanityData to 19 Oct 2026

Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding

arXiv 2609.18686 ↗First posted 16 Sep 2026cs.SDcited by 0

On the map: Speech and audio language models → Speech recognition models and training

Abstract (arXiv)

Self-supervised learning (SSL) has improved speech representations, yet performance degrades in pathological domains such as electrolaryngeal (EL) speech, and the computational footprint of SSL models limits their applicability in real-time, on-device deployment. We propose a multi-teacher knowledge distillation framework to train a lightweight, streaming content encoder that generalizes across healthy (HE) and EL speech. Two teachers are distilled progressively: a frozen SSL model providing discrete phonetic cluster targets from HE speech, and an EL-fine-tuned speech recognition model supplying continuous bottleneck feature targets. Evaluated via downstream speech recognition, our approach reduces the EL word error rate to 21.2%, compared to 39.3% for the strongest zero-shot SSL baseline. Among causal convolutional, Transformer, Conformer, and Mamba-based student architectures, a Mel-Conformer achieves the best combination of EL accuracy and computational efficiency. The final encoder contains 21.9,M parameters and runs at a real-time factor of 0.30 under ONNX Runtime on a single CPU core.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Speech recognition models and training, one of 11 directions in Speech and audio language models. Builds on 37 of its 51 references that are in the map. First posted on arXiv 16 Sep 2026.

Builds on

37 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Moonshine v2: Ergodic Streaming Encoder ASR for Latency-Critical Speech Applications12 Feb
  2. 2023Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition20 Mar
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervision6 Dec
  4. 2022Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognition30 Mar
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing26 Oct

Other references · 32

  1. 2017Attention Is All You Needcited by 61,180
  2. 2014Adam: A Method for Stochastic Optimizationcited by 38,210
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,879
  4. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,814
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,990
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 2,293
  8. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,596
  9. 2021SoundStream: An End-to-End Neural Audio Codeccited by 988
  10. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 589
  11. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 535
  12. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 412
20 more
  1. 2019On the Efficacy of Knowledge Distillationcited by 385
  2. 2024Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generationcited by 268
  3. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 235
  4. 2022FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversioncited by 105
  5. 2023AudioDec: An Open-source Streaming High-fidelity Neural Audio Codeccited by 99
  6. 2024mHuBERT-147: A Compact Multilingual HuBERT Modelcited by 73
  7. 2021A Comparison of Discrete and Soft Speech Units for Improved Voice Conversioncited by 70
  8. 2021HUI-Audio-Corpus-German: A high quality TTS datasetcited by 20
  9. 2023CML-TTS A Multilingual Dataset for Speech Synthesis in Low-Resource Languagescited by 20
  10. 2024StreamVC: Real-Time Low-Latency Voice Conversioncited by 20
  11. 2023Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learningcited by 11
  12. 2025RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Codingcited by 6
  13. 2026TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymizationcited by 6
  14. 2025Multi-Distillation from Speech and Music Representation Modelscited by 5
  15. 2022HiFi-VC: High Quality ASR-Based Voice Conversioncited by 4
  16. 2023Electrolaryngeal Speech Intelligibility Enhancement Through Robust Linguistic Encoderscited by 3
  17. 2025SynthVC: Leveraging Synthetic Data for End-to-End Low Latency Streaming Voice Conversioncited by 3
  18. 2025AdaptVC: High Quality Voice Conversion with Adaptive Learningcited by 2
  19. 2023A low latency attention module for streaming self-supervised speech representation learningcited by 1
  20. 2026Lightweight and perceptually-guided voice conversion for electro-laryngeal speechcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition
  2. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  3. 2026Learning Speech Attribute Representations via Multi-Teacher Distillation
  4. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models
  5. 2023Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition
  6. 2026S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models
  7. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation
  8. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models