mlsanityData to 14 Oct 2026

On-the-fly Routing for Zero-shot MoE Speaker Adaptation of Speech Foundation Models for Dysarthric Speech Recognition

arXiv 2505.22072 ↗First posted 28 May 2025cs.SD eess.AScited by 0

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

This paper proposes a novel MoE-based speaker adaptation framework for foundation models based dysarthric speech recognition. This approach enables zero-shot adaptation and real-time processing while incorporating domain knowledge. Speech impairment severity and gender conditioned adapter experts are dynamically combined using on-the-fly predicted speaker-dependent routing parameters. KL-divergence is used to further enforce diversity among experts and their generalization to unseen speakers. Experimental results on the UASpeech corpus suggest that on-the-fly MoE-based adaptation produces statistically significant WER reductions of up to 1.34% absolute (6.36% relative) over the unadapted baseline HuBERT/WavLM models. Consistent WER reductions of up to 2.55% absolute (11.44% relative) and RTF speedups of up to 7 times are obtained over batch-mode adaptation across varying speaker-level data quantities. The lowest published WER of 16.35% (46.77% on very low intelligibility) is obtained.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 34 of its 43 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

34 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024SAML: Speaker Adaptive Mixture of LoRA Experts for End-to-End ASR28 Jun
  2. 2023BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognition4 Oct
  3. 2022Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition17 Sep
  4. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts7 May
  5. 2020Aphasic Speech Recognition using a Mixture of Speech Intelligibility Experts25 Aug

Other references · 29

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2024Mixtral of Expertscited by 1,823
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  7. 2018Attentive Statistics Pooling for Deep Speaker Embeddingcited by 114
  8. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 90
  9. 2022Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognitioncited by 32
  10. 2023MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognitioncited by 31
  11. 2022Investigation of Data Augmentation Techniques for Disordered Speech Recognitioncited by 27
  12. 2022Exploring Efficient-tuning Methods in Self-supervised Speech Modelscited by 26
17 more
  1. 2022Speaker adaptation for Wav2vec2 based dysarthric ASRcited by 21
  2. 2022Speaker Adaptation Using Spectro-Temporal Deep Features for Dysarthric and Elderly Speech Recognitioncited by 19
  3. 2022Recent Progress in the CUHK Dysarthric Speech Recognition Systemcited by 17
  4. 2022Cross-lingual Self-Supervised Speech Representations for Improved Dysarthric Speech Recognitioncited by 17
  5. 2024Enhancing Pre-trained ASR System Fine-tuning for Dysarthric Speech Recognition using Adversarial Data Augmentationcited by 17
  6. 2022Investigating Self-supervised Pretraining Frameworks for Pathological Speech Recognitioncited by 15
  7. 2023Exploring Self-supervised Pre-trained ASR Models For Dysarthric and Elderly Speech Recognitioncited by 14
  8. 2022CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Modelscited by 13
  9. 2023Efficient Domain Adaptation for Speech Foundation Modelscited by 13
  10. 2022Exploiting Cross Domain Acoustic-to-articulatory Inverted Features For Disordered Speech Recognitioncited by 9
  11. 2024Self-supervised ASR Models and Features For Dysarthric and Elderly Speech Recognitioncited by 9
  12. 2023Use of Speech Impairment Severity for Dysarthric Speech Recognitioncited by 8
  13. 2022Exploiting Cross-domain And Cross-Lingual Ultrasound Tongue Imaging Features For Elderly And Dysarthric Speech Recognitioncited by 7
  14. 2023DuTa-VC: A Duration-aware Typical-to-atypical Voice Conversion Approach with Diffusion Probabilistic Modelcited by 6
  15. 2024Perceiver-Prompt: Flexible Speaker Adaptation in Whisper for Chinese Disordered Speech Recognitioncited by 4
  16. 2024Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptationcited by 2
  17. 2024Structured Speaker-Deficiency Adaptation of Foundation Models for Dysarthric and Elderly Speech Recognitioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Choosing a PEFT Variant for Per-Patient Dysarthric ASR: A Single-Speaker Case Study on Two ASR Bases
  2. 2025Improved Intelligibility of Dysarthric Speech using Conditional Flow Matching
  3. 2026End-to-End Simultaneous Dysarthric Speech Reconstruction with Frame-Level Adaptor and Multiple Wait-k Knowledge Distillation
  4. 2026DARS: Dysarthria-Aware Rhythm-Style Synthesis for ASR Enhancement
  5. 2026Transcript-Free Flow-Matching Text-to-Speech via Speech Feature Conditioning
  6. 2025DiffDSR: Dysarthric Speech Reconstruction Using Latent Diffusion Model
  7. 2024Training Data Augmentation for Dysarthric Automatic Speech Recognition by Text-to-Dysarthric-Speech Synthesis
  8. 2025VocalBench-DF: A Benchmark for Evaluating Speech LLM Robustness to Disfluency