mlsanityData to 14 Oct 2026

Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASR

arXiv 2602.12546 ↗First posted 13 Feb 2026eess.AS cs.AI cs.CLcited by 0

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

We present a decoder-only Conformer for automatic speech recognition (ASR) that processes speech and text in a single stack without external speech encoders or pretrained large language models (LLM). The model uses a modality-aware sparse mixture of experts (MoE): disjoint expert pools for speech and text with hard routing and top-1 selection, embedded in hybrid-causality Conformer blocks (bidirectional for speech, causal for text). Training combines CTC on speech positions with label-smoothed cross-entropy for text generation. Our 113M-parameter model consistently improves WER over a 139M AED baseline on Librispeech (2.8% vs. 3.2% test-clean; 5.6% vs. 6.0% test-other). On Common Voice 16.1 with a single multilingual model across five languages, our approach reduces average WER from 12.2% to 10.6%. To our knowledge, this is the first randomly initialized decoder-only ASR that surpasses strong AED baselines via modality-aware routing and sparse MoE, achieving better accuracy with fewer active parameters and without alignment/adaptation modules.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 27 of its 31 references that are in the map. First posted on arXiv 13 Feb 2026.

Builds on

27 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts31 Jul
  2. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts6 Jun
  3. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts7 May

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  9. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  12. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
12 more
  1. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  3. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  4. 2023Textually Pretrained Speech Language Modelscited by 101
  5. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 97
  6. 2021BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognitioncited by 88
  7. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  8. 2021Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognitioncited by 42
  9. 2024Efficient Streaming LLM for Speech Recognitioncited by 12
  10. 2024AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLMcited by 12
  11. 2023Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understandingcited by 9
  12. 2024Exploring the limits of decoder-only models trained on public speech recognition corporacited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASR
  2. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  3. 2022Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition
  4. 2025GLAD: Global-Local Aware Dynamic Mixture-of-Experts for Multi-Talker ASR
  5. 2026Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
  6. 2024U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF
  7. 2026Mixture-of-Experts with Intermediate CTC Supervision for Accented Speech Recognition
  8. 2025Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition