Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Recent advancements in multimodal large language models (MLLMs) have made significant progress in integrating information across various modalities, yet real-world applications in educational and scientific domains remain challenging. This paper introduces the Multimodal Scientific ASR (MS-ASR) task, which focuses on transcribing scientific conference videos by leveraging visual information from slides to enhance the accuracy of technical terminologies. Realized that traditional metrics like WER fall short in assessing performance accurately, prompting the proposal of severity-aware WER (SWER) that considers the content type and severity of ASR errors. We propose the Scientific Vision Augmented ASR (SciVASR) framework as a baseline method, enabling MLLMs to improve transcript quality through post-editing. Evaluations of state-of-the-art MLLMs, including GPT-4o, show a 45% improvement over speech-only baselines, highlighting the importance of multimodal information integration.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 31 references that are in the map. First posted on arXiv 16 Jun 2024.
Builds on
17 of 31 references.
References · 17
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
- 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
- 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
- 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
- 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
- 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
- 2023INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedbackcited by 21
- 2024Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understandingcited by 18
- 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16
- 2022Improving Multimodal Speech Recognition by Data Augmentation and Speech Representationscited by 13
5 more
- 2020Listen, Look and Deliberate: Visual context-aware speech recognition using pre-trained text-video representationscited by 10
- 2018Multimodal Grounding for Sequence-to-Sequence Speech Recognitioncited by 8
- 2023Revisiting Kernel Temporal Segmentation as an Adaptive Tokenizer for Long-form Video Understandingcited by 7
- 2024Toward Practical Automatic Speech Recognition and Post-Processing: a Call for Explainable Error Benchmark Guidelinecited by 4
- 2023Multimodal Speech Recognition for Language-Guided Embodied Agentscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
- 2023SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions
- 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
- 2026What We are Missing in Multimodal LLM Evaluation?
- 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding
- 2025Speech Recognition on TV Series with Video-guided Post-ASR Correction
- 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
- 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective