# Towards Pose-invariant Lip-Reading

arXiv: https://arxiv.org/abs/1911.06095 · HTML: https://mlsanity.com/paper/1911.06095/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2019.

## Built on by (most cited first, 12 of 12)

- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [SDFR: Synthetic Data for Face Recognition Competition](https://mlsanity.com/paper/2404.04580/) (2024)
- [Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape](https://mlsanity.com/paper/2204.04379/) (2022)
- [Advances and Challenges in Deep Lip Reading](https://mlsanity.com/paper/2110.07879/) (2021)
- [Expression-preserving face frontalization improves visually assisted speech processing](https://mlsanity.com/paper/2204.02810/) (2022)
- [LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition](https://mlsanity.com/paper/2501.04204/) (2025)
- [GLip: A Global-Local Integrated Progressive Framework for Robust Visual Speech Recognition](https://mlsanity.com/paper/2509.16031/) (2025)
- [Data standardization for robust lip sync](https://mlsanity.com/paper/2202.06198/) (2022)
- [InfoSyncNet: Information Synchronization Temporal Convolutional Network for Visual Speech Recognition](https://mlsanity.com/paper/2508.02460/) (2025)
- [Head-Pose-Aware Visual Speech Recognition with FiLM Modulation](https://mlsanity.com/paper/2606.00751/) (2026)
- [Personalized Korean Lipreading as Visual Speech Recognition: Transfer, Census and Adaptation on OLKAVS](https://mlsanity.com/paper/2609.28988/) (2026)
- [Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognition](https://mlsanity.com/paper/2609.29443/) (2026)

## Abstract (arXiv)

Lip-reading models have been significantly improved recently thanks to powerful deep learning architectures. However, most works focused on frontal or near frontal views of the mouth. As a consequence, lip-reading performance seriously deteriorates in non-frontal mouth views. In this work, we present a framework for training pose-invariant lip-reading models on synthetic data instead of collecting and annotating non-frontal data which is costly and tedious. The proposed model significantly outperforms previous approaches on non-frontal views while retaining the superior performance on frontal and near frontal mouth views. Specifically, we propose to use a 3D Morphable Model (3DMM) to augment LRW, an existing large-scale but mostly frontal dataset, by generating synthetic facial data in arbitrary poses. The newly derived dataset, is used to train a state-of-the-art neural network for lip-reading. We conducted a cross-database experiment for isolated word recognition on the LRS2 dataset, and reported an absolute improvement of 2.55%. The benefit of the proposed approach becomes clearer in extreme poses where an absolute improvement of up to 20.64% over the baseline is achieved.