VHASR: A Multimodal Speech Recognition System With Vision Hotwords
Not on the map. Why?
Abstract (arXiv)
The image-based multimodal automatic speech recognition (ASR) model enhances speech recognition performance by incorporating audio-related image. However, some works suggest that introducing image information to model does not help improving ASR performance. In this paper, we propose a novel approach effectively utilizing audio-related image information and set up VHASR, a multimodal speech recognition system that uses vision as hotwords to strengthen the model's speech recognition capability. Our system utilizes a dual-stream architecture, which firstly transcribes the text on the two streams separately, and then combines the outputs. We evaluate the proposed model on four datasets: Flickr8k, ADE20k, COCO, and OpenImages. The experimental results show that VHASR can effectively utilize key information in images to enhance the model's speech recognition ability. Its performance not only surpasses unimodal ASR, but also achieves SOTA among existing image-based multimodal ASR.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 1 Oct 2024.
Builds on
24 of 35 references.
References · 24
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
- 2023FunASR: A Fundamental End-to-End Speech Recognition Toolkitcited by 92
- 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 76
- 2015Deep Multimodal Semantic Embeddings for Speech and Imagescited by 56
- 2020Text-Free Image-to-Speech Synthesis Using Learned Segmental Unitscited by 50
- 2020CIF-based Collaborative Decoding for End-to-end Contextual Speech Recognitioncited by 20
- 2020SAN-M: Memory Equipped Self-Attention for End-to-End Speech Recognitioncited by 16
12 more
- 2023SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Abilitycited by 16
- 2022Improving Multimodal Speech Recognition by Data Augmentation and Speech Representationscited by 13
- 2018End-to-End Multimodal Speech Recognitioncited by 9
- 2023Achieving Timestamp Prediction While Recognizing with Non-Autoregressive End-to-End ASR Modelcited by 9
- 2018Multimodal Grounding for Sequence-to-Sequence Speech Recognitioncited by 8
- 2020Multimodal Speech Recognition with Unstructured Audio Maskingcited by 6
- 2020Looking Enhances Listening: Recovering Missing Speech Using Imagescited by 5
- 2020Fine-Grained Grounding for Multimodal Speech Recognitioncited by 5
- 2024Multi-modal Auto-regressive Modeling via Visual Wordscited by 4
- 2023Multimodal Speech Recognition for Language-Guided Embodied Agentscited by 3
- 2023Visual Information Matters for ASR Error Correctioncited by 3
- 2023ArcGPT: A Large Language Model Tailored for Real-world Archival Applicationscited by 3
Built on by
Citing papers per year: 2025: 2.
2 papers.