mlsanityData to 7 Oct 2026

MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts

arXiv 2601.10272 ↗First posted 15 Jan 2026cs.CL cs.AI cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

We present MoST (Mixture of Speech and Text), a novel multimodal large language model that seamlessly integrates speech and text processing through our proposed Modality-Aware Mixture of Experts (MAMoE) architecture. While current multimodal models typically process diverse modality representations with identical parameters, disregarding their inherent representational differences, we introduce specialized routing pathways that direct tokens to modality-appropriate experts based on input type. MAMoE simultaneously enhances modality-specific learning and cross-modal understanding through two complementary components: modality-specific expert groups that capture domain-specific patterns and shared experts that facilitate information transfer between modalities. Building on this architecture, we develop an efficient transformation pipeline that adapts the pretrained MoE language model through strategic post-training on ASR and TTS datasets, followed by fine-tuning with a carefully curated speech-text instruction dataset. A key feature of this pipeline is that it relies exclusively on fully accessible, open-source datasets to achieve strong performance and data efficiency. Comprehensive evaluations across ASR, TTS, audio language modeling, and spoken question answering benchmarks show that MoST consistently outperforms existing models of comparable parameter counts. Our ablation studies confirm that the modality-specific routing mechanism and shared experts design significantly contribute to performance gains across all tested domains. To our knowledge, MoST represents the first fully open-source speech-text LLM built on a Mixture of Experts architecture. footnoteWe release MoST model, training code, inference code, and training data at https://github.com/NUS-HPC-AI-Lab/MoST

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 25 references that are in the map. First posted on arXiv 15 Jan 2026.

Builds on

20 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs3 Mar
  2. 2024MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models17 Jul

Other references · 18

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  9. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  10. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
  11. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  12. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
6 more
  1. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 100
  2. 2023Textually Pretrained Speech Language Modelscited by 99
  3. 2023Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMcited by 92
  4. 2023Libriheavy: a 50,000 hours ASR corpus with punctuation casing and contextcited by 84
  5. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 75
  6. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Expertscited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusioncited by 0
  2. 2026Symbiotic Architecture for Post-Hoc Audio Extension of Frozen Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
  2. 2025DeepOmni: Towards Seamless and Smart Speech Interaction with Adaptive Modality-Specific MoE
  3. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques
  4. 2026Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing
  5. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
  6. 2024Model Composition for Multimodal Large Language Models
  7. 2025SMAR: Soft Modality-Aware Routing Strategy for MoE-based Multimodal Large Language Models Preserving Language Capabilities
  8. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration