# SparseVSR: Lightweight and Noise Robust Visual Speech Recognition

arXiv: https://arxiv.org/abs/2307.04552 · HTML: https://mlsanity.com/paper/2307.04552/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 10 Jul 2023.

## Builds on (most cited first, 26 of 26)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [LRS3-TED: a large-scale dataset for visual speech recognition](https://mlsanity.com/paper/1809.00496/) (2018)
- [Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection](https://mlsanity.com/paper/2012.07657/) (2020)
- [Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction](https://mlsanity.com/paper/2201.02184/) (2022)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://mlsanity.com/paper/2003.02389/) (2020)
- [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [End-to-end Audio-visual Speech Recognition with Conformers](https://mlsanity.com/paper/2102.06657/) (2021)
- [Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels](https://mlsanity.com/paper/2303.14307/) (2023)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Visual Speech Recognition for Multiple Languages in the Wild](https://mlsanity.com/paper/2202.13084/) (2022)
- [ASR is all you need: cross-modal distillation for lip reading](https://mlsanity.com/paper/1911.12747/) (2019)
- [The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training](https://mlsanity.com/paper/2202.02643/) (2022)
- [Recurrent Neural Network Transducer for Audio-Visual Speech Recognition](https://mlsanity.com/paper/1911.04890/) (2019)
- [End-to-end Audiovisual Speech Recognition](https://mlsanity.com/paper/1802.06424/) (2018)
- [Model Compression with Adversarial Robustness: A Unified Optimization Framework](https://mlsanity.com/paper/1902.03538/) (2019)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [Jointly Learning Visual and Auditory Speech Representations from Raw Data](https://mlsanity.com/paper/2212.06246/) (2022)
- [Towards Practical Lipreading with Distilled and Efficient Models](https://mlsanity.com/paper/2007.06504/) (2020)

## Built on by (most cited first, 7 of 7)

- [Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Models](https://mlsanity.com/paper/2410.07771/) (2024)
- [MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization](https://mlsanity.com/paper/2406.17614/) (2024)
- [Tailored Design of Audio-Visual Speech Recognition Models using Branchformers](https://mlsanity.com/paper/2407.06606/) (2024)
- [GLip: A Global-Local Integrated Progressive Framework for Robust Visual Speech Recognition](https://mlsanity.com/paper/2509.16031/) (2025)
- [Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions](https://mlsanity.com/paper/2502.00464/) (2025)
- [Head-Pose-Aware Visual Speech Recognition with FiLM Modulation](https://mlsanity.com/paper/2606.00751/) (2026)
- [Pose Adaptive Dynamic FiLM Modulation for Visual Speech Recognition](https://mlsanity.com/paper/2609.29443/) (2026)

## Abstract (arXiv)

Recent advances in deep neural networks have achieved unprecedented success in visual speech recognition. However, there remains substantial disparity between current methods and their deployment in resource-constrained devices. In this work, we explore different magnitude-based pruning techniques to generate a lightweight model that achieves higher performance than its dense model equivalent, especially under the presence of visual noise. Our sparse models achieve state-of-the-art results at 10% sparsity on the LRS3 dataset and outperform the dense equivalent up to 70% sparsity. We evaluate our 50% sparse model on 7 different visual noise types and achieve an overall absolute improvement of more than 2% WER compared to the dense equivalent. Our results confirm that sparse networks are more resistant to noise than dense networks.