# Lipreading with Long Short-Term Memory

arXiv: https://arxiv.org/abs/1601.08188 · HTML: https://mlsanity.com/paper/1601.08188/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2016.

## Built on by (most cited first, 17 of 17)

- [VALLR: Visual ASR Language Model for Lip Reading](https://mlsanity.com/paper/2503.21408/) (2025)
- [LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading](https://mlsanity.com/paper/2112.04748/) (2021)
- [Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillation](https://mlsanity.com/paper/2212.02782/) (2022)
- [SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory](https://mlsanity.com/paper/2108.13630/) (2021)
- [Is Lip Region-of-Interest Sufficient for Lipreading?](https://mlsanity.com/paper/2205.14295/) (2022)
- [SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks](https://mlsanity.com/paper/2303.01758/) (2023)
- [Towards a practical lip-to-speech conversion system using deep neural networks and mobile application frontend](https://mlsanity.com/paper/2104.14467/) (2021)
- [One-shot lip-based biometric authentication: extending behavioral features with authentication phrase information](https://mlsanity.com/paper/2308.06944/) (2023)
- [Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation](https://mlsanity.com/paper/2502.05758/) (2025)
- [SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer](https://mlsanity.com/paper/2505.04394/) (2025)
- [End-to-End Lip Reading in Romanian with Cross-Lingual Domain Adaptation and Lateral Inhibition](https://mlsanity.com/paper/2310.04858/) (2023)
- [Development and evaluation of a deep learning algorithm for German word recognition from lip movements](https://mlsanity.com/paper/2504.15792/) (2025)
- [Interactive decoding of words from visual speech recognition models](https://mlsanity.com/paper/2107.00692/) (2021)
- [Show Me Your Face, And I'll Tell You How You Speak](https://mlsanity.com/paper/2206.14009/) (2022)
- [Sequence-to-Sequence Multi-Modal Speech In-Painting](https://mlsanity.com/paper/2406.01321/) (2024)
- [Head-Pose-Aware Visual Speech Recognition with FiLM Modulation](https://mlsanity.com/paper/2606.00751/) (2026)
- [Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling](https://mlsanity.com/paper/2610.19559/) (2026)

## Abstract (arXiv)

Lipreading, i.e. speech recognition from visual-only recordings of a speaker's face, can be achieved with a processing pipeline based solely on neural networks, yielding significantly better accuracy than conventional methods. Feed-forward and recurrent neural network layers (namely Long Short-Term Memory; LSTM) are stacked to form a single structure which is trained by back-propagating error gradients through all the layers. The performance of such a stacked network was experimentally evaluated and compared to a standard Support Vector Machine classifier using conventional computer vision features (Eigenlips and Histograms of Oriented Gradients). The evaluation was performed on data from 19 speakers of the publicly available GRID corpus. With 51 different words to classify, we report a best word accuracy on held-out evaluation speakers of 79.6% using the end-to-end neural network-based solution (11.6% improvement over the best feature-based solution evaluated).