# Pushing the boundaries of audiovisual word recognition using Residual Networks and LSTMs

arXiv: https://arxiv.org/abs/1811.01194 · HTML: https://mlsanity.com/paper/1811.01194/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2018.

## Built on by (most cited first, 7 of 7)

- [LiRA: Learning Visual Speech Representations from Audio through Self-supervision](https://mlsanity.com/paper/2106.09171/) (2021)
- [Unsupervised Sound Localization via Iterative Contrastive Learning](https://mlsanity.com/paper/2104.00315/) (2021)
- [Training Strategies for Improved Lip-reading](https://mlsanity.com/paper/2209.01383/) (2022)
- [Advances and Challenges in Deep Lip Reading](https://mlsanity.com/paper/2110.07879/) (2021)
- [Learning Separable Hidden Unit Contributions for Speaker-Adaptive Lip-Reading](https://mlsanity.com/paper/2310.05058/) (2023)
- [Interactive decoding of words from visual speech recognition models](https://mlsanity.com/paper/2107.00692/) (2021)
- [TD3Net: A temporal densely connected multi-dilated convolutional network for lipreading](https://mlsanity.com/paper/2506.16073/) (2025)

## Abstract (arXiv)

Visual and audiovisual speech recognition are witnessing a renaissance which is largely due to the advent of deep learning methods. In this paper, we present a deep learning architecture for lipreading and audiovisual word recognition, which combines Residual Networks equipped with spatiotemporal input layers and Bidirectional LSTMs. The lipreading architecture attains 11.92% misclassification rate on the challenging Lipreading-In-The-Wild database, which is composed of excerpts from BBC-TV, each containing one of the 500 target words. Audiovisual experiments are performed using both intermediate and late integration, as well as several types and levels of environmental noise, and notable improvements over the audio-only network are reported, even in the case of clean speech. A further analysis on the utility of target word boundaries is provided, as well as on the capacity of the network in modeling the linguistic context of the target word. Finally, we examine difficult word pairs and discuss how visual information helps towards attaining higher recognition accuracy.