# Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition

arXiv: https://arxiv.org/abs/2003.03206 · HTML: https://mlsanity.com/paper/2003.03206/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2020.

## Built on by (most cited first, 30 of 30)

- [Sub-word Level Lip Reading With Visual Attention](https://mlsanity.com/paper/2110.07603/) (2021)
- [Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring](https://mlsanity.com/paper/2303.08536/) (2023)
- [Lip to Speech Synthesis with Visual Context Attentional GAN](https://mlsanity.com/paper/2204.01726/) (2022)
- [Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading](https://mlsanity.com/paper/2204.01725/) (2022)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [Training Strategies for Improved Lip-reading](https://mlsanity.com/paper/2209.01383/) (2022)
- [Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association](https://mlsanity.com/paper/2103.07293/) (2021)
- [Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video](https://mlsanity.com/paper/2204.01265/) (2022)
- [AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model](https://mlsanity.com/paper/2308.07593/) (2023)
- [Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge](https://mlsanity.com/paper/2308.09311/) (2023)
- [Multi-Temporal Lip-Audio Memory for Visual Speech Recognition](https://mlsanity.com/paper/2305.04542/) (2023)
- [LIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wild](https://mlsanity.com/paper/2311.12457/) (2023)
- [Is Lip Region-of-Interest Sufficient for Lipreading?](https://mlsanity.com/paper/2205.14295/) (2022)
- [Landmark-Guided Cross-Speaker Lip Reading with Mutual Information Regularization](https://mlsanity.com/paper/2403.16071/) (2024)
- [Advances and Challenges in Deep Lip Reading](https://mlsanity.com/paper/2110.07879/) (2021)
- [Enhancing Lip Reading with Multi-Scale Video and Multi-Encoder](https://mlsanity.com/paper/2404.05466/) (2024)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?](https://mlsanity.com/paper/2606.07435/) (2026)
- [Learning Separable Hidden Unit Contributions for Speaker-Adaptive Lip-Reading](https://mlsanity.com/paper/2310.05058/) (2023)
- [Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation](https://mlsanity.com/paper/2502.05758/) (2025)
- [SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer](https://mlsanity.com/paper/2505.04394/) (2025)
- [Spatio-Temporal Attention Mechanism and Knowledge Distillation for Lip Reading](https://mlsanity.com/paper/2108.03543/) (2021)
- [Lip-Listening: Mixing Senses to Understand Lips using Cross Modality Knowledge Distillation for Word-Based Models](https://mlsanity.com/paper/2207.05692/) (2022)
- [Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models](https://mlsanity.com/paper/2502.05766/) (2025)
- [Speaker-Adapted End-to-End Visual Speech Recognition for Continuous Spanish](https://mlsanity.com/paper/2311.12480/) (2023)
- [RAL:Redundancy-Aware Lipreading Model Based on Differential Learning with Symmetric Views](https://mlsanity.com/paper/2409.05307/) (2024)
- [Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions](https://mlsanity.com/paper/2502.00464/) (2025)
- [TD3Net: A temporal densely connected multi-dilated convolutional network for lipreading](https://mlsanity.com/paper/2506.16073/) (2025)
- [InfoSyncNet: Information Synchronization Temporal Convolutional Network for Visual Speech Recognition](https://mlsanity.com/paper/2508.02460/) (2025)
- [Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling](https://mlsanity.com/paper/2610.19559/) (2026)

## Abstract (arXiv)

Recent advances in deep learning have heightened interest among researchers in the field of visual speech recognition (VSR). Currently, most existing methods equate VSR with automatic lip reading, which attempts to recognise speech by analysing lip motion. However, human experience and psychological studies suggest that we do not always fix our gaze at each other's lips during a face-to-face conversation, but rather scan the whole face repetitively. This inspires us to revisit a fundamental yet somehow overlooked problem: can VSR models benefit from reading extraoral facial regions, i.e. beyond the lips? In this paper, we perform a comprehensive study to evaluate the effects of different facial regions with state-of-the-art VSR models, including the mouth, the whole face, the upper face, and even the cheeks. Experiments are conducted on both word-level and sentence-level benchmarks with different characteristics. We find that despite the complex variations of the data, incorporating information from extraoral facial regions, even the upper face, consistently benefits VSR performance. Furthermore, we introduce a simple yet effective method based on Cutout to learn more discriminative features for face-based VSR, hoping to maximise the utility of information encoded in different facial regions. Our experiments show obvious improvements over existing state-of-the-art methods that use only the lip region as inputs, a result we believe would probably provide the VSR community with some new and exciting insights.