mlsanityData to 7 Oct 2026

MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition

arXiv 2502.10447 ↗First posted 11 Feb 2025eess.AS cs.CL cs.LGcited by 4

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Audio-visual speech recognition (AVSR) has become critical for enhancing speech recognition in noisy environments by integrating both auditory and visual modalities. However, existing AVSR systems struggle to scale up without compromising computational efficiency. In this study, we introduce MoHAVE (Mixture of Hierarchical Audio-Visual Experts), a novel robust AVSR framework designed to address these scalability constraints. By leveraging a Mixture-of-Experts (MoE) architecture, MoHAVE activates modality-specific expert groups, ensuring dynamic adaptation to various audio-visual inputs with minimal computational overhead. Key contributions of MoHAVE include: (1) a sparse MoE framework that efficiently scales AVSR model capacity, (2) a hierarchical gating mechanism that dynamically utilizes the expert groups based on input context, enhancing adaptability and robustness, and (3) remarkable performance across robust AVSR benchmarks, including LRS3 and MuAViC transcription and translation tasks, setting a new standard for scalable speech recognition systems.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 63 of its 81 references that are in the map. First posted on arXiv 11 Feb 2025.

Builds on

63 of 81 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Robust Audiovisual Speech Recognition Models with Mixture-of-Experts19 Sep
  2. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts18 May
  3. 2023Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition12 Jul
  4. 2023Mixture-of-Expert Conformer for Streaming Multilingual ASR25 May
  5. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts6 Jun
  6. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routing23 Nov
  7. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts7 May

Other references · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2024Mixtral of Expertscited by 1,785
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
38 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  3. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  4. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  5. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  6. 2018Deep Audio-Visual Speech Recognitioncited by 318
  7. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  8. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  9. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  10. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  11. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  12. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  13. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  14. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  15. 2022StableMoE: Stable Routing Strategy for Mixture of Expertscited by 98
  16. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  17. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  18. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 71
  19. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  20. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  21. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
  22. 2021LiRA: Learning Visual Speech Representations from Audio through Self-supervisioncited by 41
  23. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  24. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  25. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  26. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36
  27. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  28. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 24
  29. 2022Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognitioncited by 23
  30. 2023Audio-Visual Efficient Conformer for Robust Speech Recognitioncited by 22
  31. 2022Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learningcited by 21
  32. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 20
  33. 2023PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Expertscited by 19
  34. 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16
  35. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  36. 2022Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognitioncited by 12
  37. 2024A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognitioncited by 12
  38. 2024XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perceptioncited by 11

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  2. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  3. 2026Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognitioncited by 1
  4. 2026VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models
  2. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs
  3. 2025MoCHA: Advanced Vision-Language Reasoning with MoE Connector and Hierarchical Group Attention
  4. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Context
  5. 2026MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free
  6. 2026MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition
  7. 2025Vision-Speech Models: Teaching Speech Models to Converse about Images
  8. 2026AVI-Bench: Toward Human-like Audio-Visual Intelligence of Omni-MLLMs