# InfoSyncNet: Information Synchronization Temporal Convolutional Network for Visual Speech Recognition

arXiv: https://arxiv.org/abs/2508.02460 · HTML: https://mlsanity.com/paper/2508.02460/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020)
- [End-to-end Audiovisual Speech Recognition](https://mlsanity.com/paper/1802.06424/) (2018)
- [Combining Residual Networks with LSTMs for Lipreading](https://mlsanity.com/paper/1703.04105/) (2017)
- [Discriminative Multi-modality Speech Recognition](https://mlsanity.com/paper/2005.05592/) (2020)
- [LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild](https://mlsanity.com/paper/1810.06990/) (2018)
- [Towards Practical Lipreading with Distilled and Efficient Models](https://mlsanity.com/paper/2007.06504/) (2020)
- [Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading](https://mlsanity.com/paper/2204.01725/) (2022)
- [Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition](https://mlsanity.com/paper/2003.03206/) (2020)
- [Lip-reading with Densely Connected Temporal Convolutional Networks](https://mlsanity.com/paper/2009.14233/) (2020)
- [Training Strategies for Improved Lip-reading](https://mlsanity.com/paper/2209.01383/) (2022)
- [Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading](https://mlsanity.com/paper/1905.02540/) (2019)
- [Audio-Visual Efficient Conformer for Robust Speech Recognition](https://mlsanity.com/paper/2301.01456/) (2023)
- [Learn an Effective Lip Reading Model without Pains](https://mlsanity.com/paper/2011.07557/) (2020)
- [Speaker-adaptive Lip Reading with User-dependent Padding](https://mlsanity.com/paper/2208.04498/) (2022)
- [Towards Pose-invariant Lip-Reading](https://mlsanity.com/paper/1911.06095/) (2019)
- [Multi-Temporal Lip-Audio Memory for Visual Speech Recognition](https://mlsanity.com/paper/2305.04542/) (2023)
- [On the role of Lip Articulation in Visual Speech Perception](https://mlsanity.com/paper/2203.10117/) (2022)

## Abstract (arXiv)

Estimating spoken content from silent videos is crucial for applications in Assistive Technology (AT) and Augmented Reality (AR). However, accurately mapping lip movement sequences in videos to words poses significant challenges due to variability across sequences and the uneven distribution of information within each sequence. To tackle this, we introduce InfoSyncNet, a non-uniform sequence modeling network enhanced by tailored data augmentation techniques. Central to InfoSyncNet is a non-uniform quantization module positioned between the encoder and decoder, enabling dynamic adjustment to the network's focus and effectively handling the natural inconsistencies in visual speech data. Additionally, multiple training strategies are incorporated to enhance the model's capability to handle variations in lighting and the speaker's orientation. Comprehensive experiments on the LRW and LRW1000 datasets confirm the superiority of InfoSyncNet, achieving new state-of-the-art accuracies of 92.0% and 60.7% Top-1 ACC. The code is available for download (see comments).