# Multimodal Grounding for Sequence-to-Sequence Speech Recognition

arXiv: https://arxiv.org/abs/1811.03865 · HTML: https://mlsanity.com/paper/1811.03865/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2018.

## Built on by (most cited first, 8 of 8)

- [AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR](https://mlsanity.com/paper/2303.16501/) (2023)
- [AVATAR: Unconstrained Audiovisual Speech Recognition](https://mlsanity.com/paper/2206.07684/) (2022)
- [Robust Audiovisual Speech Recognition Models with Mixture-of-Experts](https://mlsanity.com/paper/2409.12370/) (2024)
- [Adversarial Attacks on Speech Recognition Systems for Mission-Critical Applications: A Survey](https://mlsanity.com/paper/2202.10594/) (2022)
- [Multimodal Speech Recognition for Language-Guided Embodied Agents](https://mlsanity.com/paper/2302.14030/) (2023)
- [SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data](https://mlsanity.com/paper/2408.00624/) (2024)
- [VHASR: A Multimodal Speech Recognition System With Vision Hotwords](https://mlsanity.com/paper/2410.00822/) (2024)
- [Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR](https://mlsanity.com/paper/2406.10880/) (2024)

## Abstract (arXiv)

Humans are capable of processing speech by making use of multiple sensory modalities. For example, the environment where a conversation takes place generally provides semantic and/or acoustic context that helps us to resolve ambiguities or to recall named entities. Motivated by this, there have been many works studying the integration of visual information into the speech recognition pipeline. Specifically, in our previous work, we propose a multistep visual adaptive training approach which improves the accuracy of an audio-based Automatic Speech Recognition (ASR) system. This approach, however, is not end-to-end as it requires fine-tuning the whole model with an adaptation layer. In this paper, we propose novel end-to-end multimodal ASR systems and compare them to the adaptive approach by using a range of visual representations obtained from state-of-the-art convolutional neural networks. We show that adaptive training is effective for S2S models leading to an absolute improvement of 1.4% in word error rate. As for the end-to-end systems, although they perform better than baseline, the improvements are slightly less than adaptive training, 0.8 absolute WER reduction in single-best models. Using ensemble decoding, end-to-end models reach a WER of 15% which is the lowest score among all systems.