# End-to-End Multimodal Speech Recognition

arXiv: https://arxiv.org/abs/1804.09713 · HTML: https://mlsanity.com/paper/1804.09713/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2018.

## Built on by (most cited first, 9 of 9)

- [Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling](https://mlsanity.com/paper/2202.03543/) (2022)
- [Fast-Slow Transformer for Visually Grounding Speech](https://mlsanity.com/paper/2109.08186/) (2021)
- [OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation](https://mlsanity.com/paper/2107.00249/) (2021)
- [AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR](https://mlsanity.com/paper/2303.16501/) (2023)
- [AVATAR: Unconstrained Audiovisual Speech Recognition](https://mlsanity.com/paper/2206.07684/) (2022)
- [Multimodal Speech Recognition for Language-Guided Embodied Agents](https://mlsanity.com/paper/2302.14030/) (2023)
- [VHASR: A Multimodal Speech Recognition System With Vision Hotwords](https://mlsanity.com/paper/2410.00822/) (2024)
- [Two-Stage Augmentation and Adaptive CTC Fusion for Improved Robustness of Multi-Stream End-to-End ASR](https://mlsanity.com/paper/2102.03055/) (2021)
- [Modelling word learning and recognition using visually grounded speech](https://mlsanity.com/paper/2203.06937/) (2022)

## Abstract (arXiv)

Transcription or sub-titling of open-domain videos is still a challenging domain for Automatic Speech Recognition (ASR) due to the data's challenging acoustics, variable signal processing and the essentially unrestricted domain of the data. In previous work, we have shown that the visual channel -- specifically object and scene features -- can help to adapt the acoustic model (AM) and language model (LM) of a recognizer, and we are now expanding this work to end-to-end approaches. In the case of a Connectionist Temporal Classification (CTC)-based approach, we retain the separation of AM and LM, while for a sequence-to-sequence (S2S) approach, both information sources are adapted together, in a single model. This paper also analyzes the behavior of CTC and S2S models on noisy video data (How-To corpus), and compares it to results on the clean Wall Street Journal (WSJ) corpus, providing insight into the robustness of both approaches.