mlsanityData to 14 Oct 2026

MLLM-based Speech Recognition: When and How is Multimodality Beneficial?

arXiv 2507.19037 ↗First posted 25 Jul 2025cs.SD cs.CL cs.MMcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent advances in multi-modal large language models (MLLMs) have opened new possibilities for unified modeling of speech, text, images, and other modalities. Building on our prior work, this paper examines the conditions and model architectures under which multiple input modalities can improve automatic speech recognition (ASR) accuracy in noisy environments. Through experiments on synthetic and real-world data, we find that (1) harnessing more modalities usually improves ASR accuracy, as each modality provides complementary information, but the improvement depends on the amount of auditory noise. (2) Synchronized modalities (e.g., lip movements) are more useful at high noise levels whereas unsynchronized modalities (e.g., image context) are most helpful at moderate noise levels. (3) Higher-quality visual representations consistently improve ASR accuracy, highlighting the importance of developing more powerful visual encoders. (4) Mamba exhibits similar trends regarding the benefits of multimodality as do Transformers. (5) The input order of modalities as well as their weights in the loss function can significantly impact accuracy. These findings both offer practical insights and help to deepen our understanding of multi-modal speech recognition under challenging conditions.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 58 references that are in the map. First posted on arXiv 25 Jul 2025.

Builds on

50 of 58 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillation18 Feb
  2. 2024Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models22 Oct
  3. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks2 Aug
  4. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference21 Mar
  5. 2024VL-Mamba: Exploring State Space Models for Multimodal Learning20 Mar
  6. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  7. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  8. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities18 May

Other references · 42

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  10. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  11. 2025Large Language Diffusion Modelscited by 912
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
30 more
  1. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  2. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 419
  3. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  4. 2018Deep Audio-Visual Speech Recognitioncited by 322
  5. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  6. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  7. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  8. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  9. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  10. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  11. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  12. 2024Mamba in Speech: Towards an Alternative to Self-Attentioncited by 60
  13. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53
  14. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
  15. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  16. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 42
  17. 2019Multi-Moments in Time: Learning and Interpreting Models for Multi-Action Video Understandingcited by 37
  18. 2021Perceptual Score: What Data Modalities Does Your Model Perceive?cited by 35
  19. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 30
  20. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  21. 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16
  22. 2022AVATAR: Unconstrained Audiovisual Speech Recognitioncited by 13
  23. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  24. 2025Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representationscited by 9
  25. 2024SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognitioncited by 7
  26. 2024MMGER: Multi-modal and Multi-granularity Generative Error Correction with LLM for Joint Accent and Speech Recognitioncited by 6
  27. 2024Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processingcited by 5
  28. 2024Image First or Text First? Optimising the Sequencing of Modalities in Large Language Model Prompting and Reasoning Taskscited by 3
  29. 2024Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?cited by 2
  30. 2025Gesture-Aware Zero-Shot Speech Recognition for Patients with Language Disorderscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reorderingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness
  2. 2024A Survey on Evaluation of Multimodal Large Language Models
  3. 2024Model Composition for Multimodal Large Language Models
  4. 2025Beyond Modality Limitations: A Unified MLLM Approach to Automated Speaking Assessment with Effective Curriculum Learning
  5. 2026Multimodal Large Language Models as Image Classifiers
  6. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  7. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
  8. 2024A Survey on Benchmarks of Multimodal Large Language Models