# Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding

arXiv: https://arxiv.org/abs/2609.18686 · HTML: https://mlsanity.com/paper/2609.18686/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Speech recognition models and training, one of 11 directions in Speech and audio language models.
- Builds on 37 of its 51 references that are in the map.
- First posted on arXiv 16 Sep 2026.

## Placed in

- Speech and audio language models / [Speech recognition models and training](https://mlsanity.com/speech/speech-recognition-models-and-training/)

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [On the Efficacy of Knowledge Distillation](https://mlsanity.com/paper/1910.01348/) (2019)
- [Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation](https://mlsanity.com/paper/2407.05361/) (2024)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion](https://mlsanity.com/paper/2210.15418/) (2022)
- [AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec](https://mlsanity.com/paper/2305.16608/) (2023)
- [mHuBERT-147: A Compact Multilingual HuBERT Model](https://mlsanity.com/paper/2406.06371/) (2024)
- [A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion](https://mlsanity.com/paper/2111.02392/) (2021)
- [Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2203.16082/) (2022)
- [HUI-Audio-Corpus-German: A high quality TTS dataset](https://mlsanity.com/paper/2106.06309/) (2021)
- [CML-TTS A Multilingual Dataset for Speech Synthesis in Low-Resource Languages](https://mlsanity.com/paper/2306.10097/) (2023)
- [StreamVC: Real-Time Low-Latency Voice Conversion](https://mlsanity.com/paper/2401.03078/) (2024)
- [Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition](https://mlsanity.com/paper/2303.10917/) (2023)
- [Joint Prediction and Denoising for Large-scale Multilingual Self-supervised Learning](https://mlsanity.com/paper/2309.15317/) (2023)
- [RT-VC: Real-Time Zero-Shot Voice Conversion with Speech Articulatory Coding](https://mlsanity.com/paper/2506.10289/) (2025)
- [TVTSyn: Content-Synchronous Time-Varying Timbre for Streaming Voice Conversion and Anonymization](https://mlsanity.com/paper/2602.09389/) (2026)
- [Multi-Distillation from Speech and Music Representation Models](https://mlsanity.com/paper/2506.07237/) (2025)

## Abstract (arXiv)

Self-supervised learning (SSL) has improved speech representations, yet performance degrades in pathological domains such as electrolaryngeal (EL) speech, and the computational footprint of SSL models limits their applicability in real-time, on-device deployment. We propose a multi-teacher knowledge distillation framework to train a lightweight, streaming content encoder that generalizes across healthy (HE) and EL speech. Two teachers are distilled progressively: a frozen SSL model providing discrete phonetic cluster targets from HE speech, and an EL-fine-tuned speech recognition model supplying continuous bottleneck feature targets. Evaluated via downstream speech recognition, our approach reduces the EL word error rate to 21.2%, compared to 39.3% for the strongest zero-shot SSL baseline. Among causal convolutional, Transformer, Conformer, and Mamba-based student architectures, a Mel-Conformer achieves the best combination of EL accuracy and computational efficiency. The final encoder contains 21.9,M parameters and runs at a real-time factor of 0.30 under ONNX Runtime on a single CPU core.