mlsanityData to 7 Oct 2026

Handling Trade-Offs in Speech Separation with Sparsely-Gated Mixture of Experts

arXiv 2211.06493 ↗First posted 11 Nov 2022eess.AS cs.SD eess.SPcited by 2

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Employing a monaural speech separation (SS) model as a front-end for automatic speech recognition (ASR) involves balancing two kinds of trade-offs. First, while a larger model improves the SS performance, it also requires a higher computational cost. Second, an SS model that is more optimized for handling overlapped speech is likely to introduce more processing artifacts in non-overlapped-speech regions. In this paper, we address these trade-offs with a sparsely-gated mixture-of-experts (MoE) architecture. Comprehensive evaluation results obtained using both simulated and real meeting recordings show that our proposed sparsely-gated MoE SS model achieves superior separation capabilities with less speech distortion, while involving only a marginal run-time cost increase.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 27 of its 40 references that are in the map. First posted on arXiv 11 Nov 2022.

Builds on

27 of 40 references.

References · 27

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  7. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 213
  8. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  9. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  10. 2020Attention is All You Need in Speech Separationcited by 184
  11. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  12. 2017TasNet: time-domain audio separation network for real-time, single-channel speech separationcited by 115
15 more
  1. 2022ICASSP 2022 Deep Noise Suppression Challengecited by 57
  2. 2020Continuous Speech Separation with Conformercited by 36
  3. 2022Investigating self-supervised learning for speech enhancement and separationcited by 35
  4. 2020Integration of speech separation, diarization, and recognition for multi-speaker meetings: System description, comparison, and analysiscited by 34
  5. 2021Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphonecited by 33
  6. 2022How Bad Are Artifacts?: Analyzing the Impact of Speech Enhancement Errors on ASRcited by 27
  7. 2018Recognizing Overlapped Speech in Meetings: A Multichannel Separation Approach Using Neural Networkscited by 23
  8. 2019Advances in Online Audio-Visual Meeting Transcriptioncited by 23
  9. 2022End-to-End Integration of Speech Recognition, Speech Enhancement, and Self-Supervised Learning Representationcited by 17
  10. 2021VarArray: Array-Geometry-Agnostic Continuous Speech Separationcited by 10
  11. 2021Investigation of Practical Aspects of Single Channel Speech Separation for ASRcited by 8
  12. 2022Learning to Enhance or Not: Neural Network-Based Switching of Enhanced and Observed Signals for Overlapping Speech Recognitioncited by 7
  13. 2022Speech separation with large-scale self-supervised learningcited by 5
  14. 2021Should We Always Separate?: Switching Between Enhanced and Observed Signals for Overlapping Speech Recognitioncited by 4
  15. 2022Leveraging Real Conversational Data for Multi-Channel Continuous Speech Separationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Improving Practical Aspects of End-to-End Multi-Talker Speech Recognition for Online and Offline Scenarioscited by 0
  2. 2026TF-MoE: Time-Frequency Mixture-of-Experts for Efficient Speech Separationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition
  2. 2026Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASR
  3. 2025Beyond Hard Sharing: Efficient Multi-Task Speech-to-Text Modeling with Supervised Mixture of Experts
  4. 2025GLAD: Global-Local Aware Dynamic Mixture-of-Experts for Multi-Talker ASR
  5. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition
  6. 2026SEAL: Mixture-Closed Additive Reconstruction and Refinement-Aware Expert Routing for Efficient Speech Separation
  7. 2022Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition
  8. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASR