mlsanityData to 14 Oct 2026

UME: Upcycling Mixture-of-Experts for Scalable and Efficient Automatic Speech Recognition

arXiv 2412.17507 ↗First posted 23 Dec 2024eess.AScited by 1

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Recent advancements in scaling up models have significantly improved performance in Automatic Speech Recognition (ASR) tasks. However, training large ASR models from scratch remains costly. To address this issue, we introduce UME, a novel method that efficiently Upcycles pretrained dense ASR checkpoints into larger Mixture-of-Experts (MoE) architectures. Initially, feed-forward networks are converted into MoE layers. By reusing the pretrained weights, we establish a robust foundation for the expanded model, significantly reducing optimization time. Then, layer freezing and expert balancing strategies are employed to continue training the model, further enhancing performance. Experiments on a mixture of 170k-hour Mandarin and English datasets show that UME: 1) surpasses the pretrained baseline by a margin of 11.9% relative error rate reduction while maintaining comparable latency; 2) reduces training time by up to 86.7% and achieves superior accuracy compared to training models of the same size from scratch.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 38 of its 42 references that are in the map. First posted on arXiv 23 Dec 2024.

Builds on

38 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF25 Apr
  2. 2023BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognition4 Oct
  3. 2023Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition12 Jul
  4. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASR25 May
  5. 20223M: Multi-loss, Multi-path and Multi-level Neural Networks for speech recognition7 Apr
  6. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routing23 Nov
  7. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts7 May

Other references · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2024Mixtral of Expertscited by 1,823
  9. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  10. 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 399
  11. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  12. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
19 more
  1. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  2. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
  3. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 187
  4. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  5. 2022Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognitioncited by 144
  6. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
  7. 2023End-to-End Speech Recognition: A Surveycited by 129
  8. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  9. 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 95
  10. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  11. 2021AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenariocited by 60
  12. 2021Exploiting Adapters for Cross-lingual Low-resource Speech Recognitioncited by 35
  13. 2021Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speechcited by 31
  14. 2023MoLE : Mixture of Language Experts for Multi-Lingual Automatic Speech Recognitioncited by 31
  15. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  16. 2023Sparsely Shared LoRA on Whisper for Child Speech Recognitioncited by 18
  17. 2024SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezingcited by 12
  18. 2020Incremental Learning for End-to-End Automatic Speech Recognitioncited by 10
  19. 2024MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Taskscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026From Self-Supervised Speech Models to Mixture-of-Experts for Robust Anti-Spoofingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2025Beyond Hard Sharing: Efficient Multi-Task Speech-to-Text Modeling with Supervised Mixture of Experts
  3. 2026Qwen-Audio-3.0-ASR Technical Report
  4. 2024BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts
  5. 2026SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling
  6. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Models
  7. 2024A Survey on Mixture of Experts in Large Language Models
  8. 2024Upcycling Large Language Models into Mixture of Experts