# Multimodal Speech Recognition with Unstructured Audio Masking

arXiv: https://arxiv.org/abs/2010.08642 · HTML: https://mlsanity.com/paper/2010.08642/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2020.

## Built on by (most cited first, 6 of 6)

- [Text is no more Enough! A Benchmark for Profile-based Spoken Language Understanding](https://mlsanity.com/paper/2112.11953/) (2021)
- [MMLatch: Bottom-up Top-down Fusion for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2201.09828/) (2022)
- [Adversarial Attacks on Speech Recognition Systems for Mission-Critical Applications: A Survey](https://mlsanity.com/paper/2202.10594/) (2022)
- [Multimodal Speech Recognition for Language-Guided Embodied Agents](https://mlsanity.com/paper/2302.14030/) (2023)
- [VHASR: A Multimodal Speech Recognition System With Vision Hotwords](https://mlsanity.com/paper/2410.00822/) (2024)
- [VILAS: Exploring the Effects of Vision and Language Context in Automatic Speech Recognition](https://mlsanity.com/paper/2305.19972/) (2023)

## Abstract (arXiv)

Visual context has been shown to be useful for automatic speech recognition (ASR) systems when the speech signal is noisy or corrupted. Previous work, however, has only demonstrated the utility of visual context in an unrealistic setting, where a fixed set of words are systematically masked in the audio. In this paper, we simulate a more realistic masking scenario during model training, called RandWordMask, where the masking can occur for any word segment. Our experiments on the Flickr 8K Audio Captions Corpus show that multimodal ASR can generalize to recover different types of masked words in this unstructured masking setting. Moreover, our analysis shows that our models are capable of attending to the visual signal when the audio signal is corrupted. These results show that multimodal ASR systems can leverage the visual signal in more generalized noisy scenarios.