# GLip: A Global-Local Integrated Progressive Framework for Robust Visual Speech Recognition

arXiv: https://arxiv.org/abs/2509.16031 · HTML: https://mlsanity.com/paper/2509.16031/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 19 Sep 2025.

## Builds on (most cited first, 24 of 24)

- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection](https://mlsanity.com/paper/2012.07657/) (2020)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels](https://mlsanity.com/paper/2303.14307/) (2023)
- [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [Robust Self-Supervised Audio-Visual Speech Recognition](https://mlsanity.com/paper/2201.01763/) (2022)
- [Discriminative Multi-modality Speech Recognition](https://mlsanity.com/paper/2005.05592/) (2020)
- [LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild](https://mlsanity.com/paper/1810.06990/) (2018)
- [Jointly Learning Visual and Auditory Speech Representations from Raw Data](https://mlsanity.com/paper/2212.06246/) (2022)
- [LiRA: Learning Visual Speech Representations from Audio through Self-supervision](https://mlsanity.com/paper/2106.09171/) (2021)
- [Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring](https://mlsanity.com/paper/2303.08536/) (2023)
- [Audio-visual Recognition of Overlapped speech for the LRS2 dataset](https://mlsanity.com/paper/2001.01656/) (2020)
- [Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning](https://mlsanity.com/paper/2212.05301/) (2022)
- [SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization](https://mlsanity.com/paper/2406.12233/) (2024)
- [Towards Pose-invariant Lip-Reading](https://mlsanity.com/paper/1911.06095/) (2019)
- [Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2305.09212/) (2023)
- [Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition](https://mlsanity.com/paper/2306.10563/) (2023)
- [SparseVSR: Lightweight and Noise Robust Visual Speech Recognition](https://mlsanity.com/paper/2307.04552/) (2023)
- [Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition](https://mlsanity.com/paper/2407.03563/) (2024)
- [Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation](https://mlsanity.com/paper/2504.18539/) (2025)
- [LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition](https://mlsanity.com/paper/2501.04204/) (2025)

## Built on by (most cited first, 2 of 2)

- [Head-Pose-Aware Visual Speech Recognition with FiLM Modulation](https://mlsanity.com/paper/2606.00751/) (2026)
- [Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognition](https://mlsanity.com/paper/2609.29443/) (2026)

## Abstract (arXiv)

Visual speech recognition (VSR), also known as lip reading, is the task of recognizing speech from silent video. Despite significant advancements in VSR over recent decades, most existing methods pay limited attention to real-world visual challenges such as illumination variations, occlusions, blurring, and pose changes. To address these challenges, we propose GLip, a Global-Local Integrated Progressive framework designed for robust VSR. GLip is built upon two key insights: (i) learning an initial coarse alignment between visual features across varying conditions and corresponding speech content facilitates the subsequent learning of precise visual-to-speech mappings in challenging environments; (ii) under adverse conditions, certain local regions (e.g., non-occluded areas) often exhibit more discriminative cues for lip reading than global features. To this end, GLip introduces a dual-path feature extraction architecture that integrates both global and local features within a two-stage progressive learning framework. In the first stage, the model learns to align both global and local visual features with corresponding acoustic speech units using easily accessible audio-visual data, establishing a coarse yet semantically robust foundation. In the second stage, we introduce a Contextual Enhancement Module (CEM) to dynamically integrate local features with relevant global context across both spatial and temporal dimensions, refining the coarse representations into precise visual-speech mappings. Our framework uniquely exploits discriminative local regions through a progressive learning strategy, demonstrating enhanced robustness against various visual challenges and consistently outperforming existing methods on the LRS2 and LRS3 benchmarks. We further validate its effectiveness on a newly introduced challenging Mandarin dataset.