# Deep word embeddings for visual speech recognition

arXiv: https://arxiv.org/abs/1710.11201 · HTML: https://mlsanity.com/paper/1710.11201/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2017.

## Built on by (most cited first, 3 of 3)

- [Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification](https://mlsanity.com/paper/2302.11254/) (2023)
- [Exploring Deep Learning for Joint Audio-Visual Lip Biometrics](https://mlsanity.com/paper/2104.08510/) (2021)
- [Designing Practical Models for Isolated Word Visual Speech Recognition](https://mlsanity.com/paper/2508.17894/) (2025)

## Abstract (arXiv)

In this paper we present a deep learning architecture for extracting word embeddings for visual speech recognition. The embeddings summarize the information of the mouth region that is relevant to the problem of word recognition, while suppressing other types of variability such as speaker, pose and illumination. The system is comprised of a spatiotemporal convolutional layer, a Residual Network and bidirectional LSTMs and is trained on the Lipreading in-the-wild database. We first show that the proposed architecture goes beyond state-of-the-art on closed-set word identification, by attaining 11.92% error rate on a vocabulary of 500 words. We then examine the capacity of the embeddings in modelling words unseen during training. We deploy Probabilistic Linear Discriminant Analysis (PLDA) to model the embeddings and perform low-shot learning experiments on words unseen during training. The experiments demonstrate that word-level visual speech recognition is feasible even in cases where the target words are not included in the training set.