# VHASR: A Multimodal Speech Recognition System With Vision Hotwords

arXiv: https://arxiv.org/abs/2410.00822 · HTML: https://mlsanity.com/paper/2410.00822/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2305.11013/) (2023)
- [CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition](https://mlsanity.com/paper/1905.11235/) (2019)
- [Deep Multimodal Semantic Embeddings for Speech and Images](https://mlsanity.com/paper/1511.03690/) (2015)
- [Text-Free Image-to-Speech Synthesis Using Learned Segmental Units](https://mlsanity.com/paper/2012.15454/) (2020)
- [CIF-based Collaborative Decoding for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2012.09466/) (2020)
- [SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Ability](https://mlsanity.com/paper/2308.03266/) (2023)
- [SAN-M: Memory Equipped Self-Attention for End-to-End Speech Recognition](https://mlsanity.com/paper/2006.01713/) (2020)
- [Improving Multimodal Speech Recognition by Data Augmentation and Speech Representations](https://mlsanity.com/paper/2204.13206/) (2022)
- [Achieving Timestamp Prediction While Recognizing with Non-Autoregressive End-to-End ASR Model](https://mlsanity.com/paper/2301.12343/) (2023)
- [End-to-End Multimodal Speech Recognition](https://mlsanity.com/paper/1804.09713/) (2018)
- [Multimodal Grounding for Sequence-to-Sequence Speech Recognition](https://mlsanity.com/paper/1811.03865/) (2018)
- [Multimodal Speech Recognition with Unstructured Audio Masking](https://mlsanity.com/paper/2010.08642/) (2020)
- [Looking Enhances Listening: Recovering Missing Speech Using Images](https://mlsanity.com/paper/2002.05639/) (2020)
- [Fine-Grained Grounding for Multimodal Speech Recognition](https://mlsanity.com/paper/2010.02384/) (2020)
- [Multi-modal Auto-regressive Modeling via Visual Words](https://mlsanity.com/paper/2403.07720/) (2024)
- [Multimodal Speech Recognition for Language-Guided Embodied Agents](https://mlsanity.com/paper/2302.14030/) (2023)
- [Visual Information Matters for ASR Error Correction](https://mlsanity.com/paper/2303.10160/) (2023)
- [ArcGPT: A Large Language Model Tailored for Real-world Archival Applications](https://mlsanity.com/paper/2307.14852/) (2023)

## Built on by (most cited first, 2 of 2)

- [Locate-and-Focus: Enhancing Terminology Translation in Speech Language Models](https://mlsanity.com/paper/2507.18263/) (2025)
- [PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMs](https://mlsanity.com/paper/2509.19745/) (2025)

## Abstract (arXiv)

The image-based multimodal automatic speech recognition (ASR) model enhances speech recognition performance by incorporating audio-related image. However, some works suggest that introducing image information to model does not help improving ASR performance. In this paper, we propose a novel approach effectively utilizing audio-related image information and set up VHASR, a multimodal speech recognition system that uses vision as hotwords to strengthen the model's speech recognition capability. Our system utilizes a dual-stream architecture, which firstly transcribes the text on the two streams separately, and then combines the outputs. We evaluate the proposed model on four datasets: Flickr8k, ADE20k, COCO, and OpenImages. The experimental results show that VHASR can effectively utilize key information in images to enhance the model's speech recognition ability. Its performance not only surpasses unimodal ASR, but also achieves SOTA among existing image-based multimodal ASR.