mlsanityData to 14 Oct 2026

Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach

arXiv 2505.14336 ↗First posted 20 May 2025eess.AS cs.CV cs.MMcited by 10

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Audio-Visual Speech Recognition (AVSR) enhances robustness in noisy environments by integrating visual cues. While recent advances integrate Large Language Models (LLMs) into AVSR, their high computational cost hinders deployment in resource-constrained settings. To address this, we propose Llama-SMoP, an efficient Multimodal LLM that employs a Sparse Mixture of Projectors (SMoP) module to scale model capacity without increasing inference costs. By incorporating sparsely-gated mixture-of-experts (MoE) projectors, Llama-SMoP enables the use of smaller LLMs while maintaining strong performance. We explore three SMoP configurations and show that Llama-SMoP DEDR (Disjoint-Experts, Disjoint-Routers), which uses modality-specific routers and experts, achieves superior performance on ASR, VSR, and AVSR tasks. Ablation studies confirm its effectiveness in expert activation, scalability, and noise robustness.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 53 of its 58 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

53 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition11 Feb
  2. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts9 May
  3. 2024Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapters1 Feb

Other references · 50

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  12. 2023VILA: On Pre-training for Visual Language Modelscited by 692
38 more
  1. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  2. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  3. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  4. 2018Deep Audio-Visual Speech Recognitioncited by 322
  5. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  6. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  7. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  8. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  9. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  10. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
  11. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  12. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  13. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  14. 2023End-to-End Speech Recognition: A Surveycited by 129
  15. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  16. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 109
  17. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  18. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  19. 2024Mixture of A Million Expertscited by 72
  20. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 64
  21. 2024Dense Connector for MLLMscited by 49
  22. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 47
  23. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 45
  24. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 42
  25. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 41
  26. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 39
  27. 2024Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translationcited by 36
  28. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 31
  29. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 30
  30. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  31. 2022Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognitioncited by 24
  32. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understandingcited by 23
  33. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  34. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 21
  35. 2024SSR: Alignment-Aware Modality Connector for Speech Language Modelscited by 16
  36. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 15
  37. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  38. 2023Parameter-Efficient Transfer Learning of Audio Spectrogram Transformerscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMscited by 11
  2. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  3. 2025Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Modelscited by 4
  4. 2026HumanOmni-Speaker: Identifying Who said What and Whencited by 3
  5. 2026When AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapsecited by 2
  6. 2025Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypothesescited by 1
  7. 2026Dynamic Multi-Expert Projectors with Stabilized Routing for Multilingual Speech Recognitioncited by 1
  8. 2026Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognitioncited by 1
  9. 2026Entropy-Aware Domain-Routed Mixture-of-Experts Speech-LLM Framework: A Case Study of Multi-Domain Child-Adult ASRcited by 1
  10. 2026VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR
  2. 2024Robust Audiovisual Speech Recognition Models with Mixture-of-Experts
  3. 2025Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
  4. 2026Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling
  5. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models
  6. 2026Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
  7. 2025Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness
  8. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models