mlsanityData to 7 Oct 2026

Boosting Code-Switching ASR with Mixture of Experts Enhanced Speech-Conditioned LLM

arXiv 2409.15905 ↗First posted 24 Sep 2024cs.SD cs.AI eess.AScited by 9

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

In this paper, we introduce a speech-conditioned Large Language Model (LLM) integrated with a Mixture of Experts (MoE) based connector to address the challenge of Code-Switching (CS) in Automatic Speech Recognition (ASR). Specifically, we propose an Insertion and Deletion of Interruption Token (IDIT) mechanism for better transfer text generation ability of LLM to speech recognition task. We also present a connecter with MoE architecture that manages multiple languages efficiently. To further enhance the collaboration of multiple experts and leverage the understanding capabilities of LLM, we propose a two-stage progressive training strategy: 1) The connector is unfrozen and trained with language-specialized experts to map speech representations to the text space. 2) The connector and LLM LoRA adaptor are trained with the proposed IDIT mechanism and all experts are activated to learn general representations. Experimental results demonstrate that our method significantly outperforms state-of-the-art models, including end-to-end and large-scale audio-language models.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 22 of its 28 references that are in the map. First posted on arXiv 24 Sep 2024.

Builds on

22 of 28 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Enhancing Code-Switching Speech Recognition with LID-Based Collaborative Mixture of Experts Model3 Sep
  2. 2024SC-MoE: Switch Conformer Mixture of Experts for Unified Streaming and Non-streaming Code-Switching ASR26 Jun
  3. 2023BA-MoE: Boundary-Aware Mixture-of-Experts Adapter for Code-Switching Speech Recognition4 Oct

Other references · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  9. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  10. 2024Qwen2-Audio Technical Reportcited by 543
  11. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  12. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
7 more
  1. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  2. 2024FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMscited by 118
  3. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 77
  4. 2024Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognitioncited by 61
  5. 2020The ASRU 2019 Mandarin-English Code-Switching Speech Recognition Challenge: Open Datasets, Tracks, Methods and Resultscited by 43
  6. 2021Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?cited by 38
  7. 2024Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasetscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 6.

9 papers.

  1. 2025Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalitiescited by 4
  2. 2026Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMscited by 2
  3. 2026Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsamplingcited by 2
  4. 2024A Comparative Study of LLM-based ASR and Whisper in Low Resource and Code Switching Scenariocited by 1
  5. 2025GLAD: Global-Local Aware Dynamic Mixture-of-Experts for Multi-Talker ASRcited by 1
  6. 2026A Language-Agnostic Hierarchical LoRA-MoE Architecture for CTC-based Multilingual ASRcited by 1
  7. 2026LLMs-Integrated Automatic Hate Speech Recognition Using Controllable Text Generation Modelscited by 0
  8. 2026Towards Truly Multilingual ASR: Generalizing Code-Switching ASR to Unseen Language Pairscited by 0
  9. 2026Dual-Form ASR: Semantics-Aware Inverse Text Normalization for Chinese Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dynamic Language Group-Based MoE: Enhancing Code-Switching Speech Recognition with Hierarchical Routing
  2. 2023Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition
  3. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  4. 2024CAMEL: Cross-Attention Enhanced Mixture-of-Experts and Language Bias for Code-Switching Speech Recognition
  5. 2025Towards Fine-Grained Code-Switch Speech Translation with Semantic Space Alignment
  6. 2024HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Models
  7. 2021Building a great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition
  8. 2025Beyond Hard Sharing: Efficient Multi-Task Speech-to-Text Modeling with Supervised Mixture of Experts