# Towards Practical Lipreading with Distilled and Efficient Models

arXiv: https://arxiv.org/abs/2007.06504 · HTML: https://mlsanity.com/paper/2007.06504/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2020.

## Built on by (most cited first, 30 of 45)

- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels](https://mlsanity.com/paper/2303.14307/) (2023)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [Self-Supervised Video Forensics by Audio-Visual Anomaly Detection](https://mlsanity.com/paper/2301.01767/) (2023)
- [LiRA: Learning Visual Speech Representations from Audio through Self-supervision](https://mlsanity.com/paper/2106.09171/) (2021)
- [Delving into Sequential Patches for Deepfake Detection](https://mlsanity.com/paper/2207.02803/) (2022)
- [Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading](https://mlsanity.com/paper/2204.01725/) (2022)
- [Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing](https://mlsanity.com/paper/2402.15151/) (2024)
- [Learning to Dub Movies via Hierarchical Prosody Models](https://mlsanity.com/paper/2212.04054/) (2022)
- [Training Strategies for Improved Lip-reading](https://mlsanity.com/paper/2209.01383/) (2022)
- [Self-supervised Transformer for Deepfake Detection](https://mlsanity.com/paper/2203.01265/) (2022)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [StyleDubber: Towards Multi-Scale Style Learning for Movie Dubbing](https://mlsanity.com/paper/2402.12636/) (2024)
- [AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model](https://mlsanity.com/paper/2308.07593/) (2023)
- [Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge](https://mlsanity.com/paper/2308.09311/) (2023)
- [Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognition](https://mlsanity.com/paper/2302.08102/) (2023)
- [SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization](https://mlsanity.com/paper/2406.12233/) (2024)
- [Multi-Temporal Lip-Audio Memory for Visual Speech Recognition](https://mlsanity.com/paper/2305.04542/) (2023)
- [Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation](https://mlsanity.com/paper/2401.09802/) (2024)
- [SparseVSR: Lightweight and Noise Robust Visual Speech Recognition](https://mlsanity.com/paper/2307.04552/) (2023)
- [FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing](https://mlsanity.com/paper/2505.01263/) (2025)
- [Advances and Challenges in Deep Lip Reading](https://mlsanity.com/paper/2110.07879/) (2021)
- [Expression-preserving face frontalization improves visually assisted speech processing](https://mlsanity.com/paper/2204.02810/) (2022)
- [Audio-Visual Speech Separation in Noisy Environments with a Lightweight Iterative Model](https://mlsanity.com/paper/2306.00160/) (2023)
- [LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition](https://mlsanity.com/paper/2501.04204/) (2025)
- [Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification](https://mlsanity.com/paper/2302.11254/) (2023)
- [Word-level Persian Lipreading Dataset](https://mlsanity.com/paper/2304.04068/) (2023)
- [Zero-shot personalized lip-to-speech synthesis with face image based voice control](https://mlsanity.com/paper/2305.14359/) (2023)
- [MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization](https://mlsanity.com/paper/2406.17614/) (2024)

## Abstract (arXiv)

Lipreading has witnessed a lot of progress due to the resurgence of neural networks. Recent works have placed emphasis on aspects such as improving performance by finding the optimal architecture or improving generalization. However, there is still a significant gap between the current methodologies and the requirements for an effective deployment of lipreading in practical scenarios. In this work, we propose a series of innovations that significantly bridge that gap: first, we raise the state-of-the-art performance by a wide margin on LRW and LRW-1000 to 88.5% and 46.6%, respectively using self-distillation. Secondly, we propose a series of architectural changes, including a novel Depthwise Separable Temporal Convolutional Network (DS-TCN) head, that slashes the computational cost to a fraction of the (already quite efficient) original model. Thirdly, we show that knowledge distillation is a very effective tool for recovering performance of the lightweight models. This results in a range of models with different accuracy-efficiency trade-offs. However, our most promising lightweight models are on par with the current state-of-the-art while showing a reduction of 8.2x and 3.9x in terms of computational cost and number of parameters, respectively, which we hope will enable the deployment of lipreading models in practical applications.