mlsanityData to 7 Oct 2026

TF-MoE: Time-Frequency Mixture-of-Experts for Efficient Speech Separation

arXiv 2606.29575 ↗First posted 28 Jun 2026cs.SD cs.AIcited by 0

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Recent advances in speech separation (SS) have led to compact front-end models with small parameter sizes, yet their high computational cost remains a major barrier for deployment on edge devices. To address this, we propose TF-MoE, a sparse Mixture-of-Experts (MoE) framework that enhances model capacity with almost no increase in inference cost. Our method introduces dynamic expert specialization in time and frequency dimensions through alternating time-wise and frequency-wise MoE modules, each dynamically selecting experts per frame or mel band. Built upon a mel-band-splitting Conformer backbone, TF-MoE achieves strong performance on SS tasks under low-compute settings. Experimental results demonstrate that TF-MoE consistently improves separation performance under computation cost constraints, outperforming BSRNN by +3.8 dB SDR on Libri2Mix with comparable 4.1 GMACs/s inference cost. This positions TF-MoE as a promising candidate for edge-device deployment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 30 of its 41 references that are in the map. First posted on arXiv 28 Jun 2026.

Builds on

30 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling28 Sep
  2. 2022Handling Trade-Offs in Speech Separation with Sparsely-Gated Mixture of Experts11 Nov
  3. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition10 Dec
  4. 2017Speech Enhancement using a Deep Mixture of Experts27 Mar

Other references · 26

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  5. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  6. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 213
  7. 2016Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separationcited by 189
  8. 2020Attention is All You Need in Speech Separationcited by 184
  9. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 182
  10. 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 154
  11. 2022Speech Enhancement with Score-Based Generative Models in the Complex STFT Domaincited by 102
  12. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
14 more
  1. 2023ICASSP 2023 Deep Noise Suppression Challengecited by 62
  2. 2020Sudo rm -rf: Efficient Networks for Universal Audio Source Separationcited by 50
  3. 2022TF-GridNet: Making Time-Frequency Domain Models Great Again for Monaural Speaker Separationcited by 48
  4. 2020TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aidscited by 41
  5. 2024The VoicePrivacy 2024 Challenge Evaluation Plancited by 39
  6. 2022An efficient encoder-decoder architecture with top-down attention for speech separationcited by 38
  7. 2021Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Networkcited by 32
  8. 2024URGENT Challenge: Universality, Robustness, and Generalizability For Speech Enhancementcited by 17
  9. 2022SkiM: Skipping Memory LSTM for Low-Latency Real-Time Continuous Speech Separationcited by 16
  10. 2025Interspeech 2025 URGENT Speech Enhancement Challengecited by 12
  11. 2024TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separationcited by 10
  12. 2024Beyond Performance Plateaus: A Comprehensive Study on Scalability in Speech Enhancementcited by 4
  13. 2023CheapNET: Improving Light-weight speech enhancement network by projected loss functioncited by 1
  14. 2025Stack Less, Repeat More: A Block Reusing Approach for Progressive Speech Enhancementcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SEAL: Mixture-Closed Additive Reconstruction and Refinement-Aware Expert Routing for Efficient Speech Separation
  2. 2024MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  3. 2025MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems
  4. 2024U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF
  5. 2025Beyond Hard Sharing: Efficient Multi-Task Speech-to-Text Modeling with Supervised Mixture of Experts
  6. 2025EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models
  7. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques
  8. 2026OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit