# SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory arXiv: https://arxiv.org/abs/2108.13630 · HTML: https://mlsanity.com/paper/2108.13630/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 44 references that are in the map. - First posted on arXiv 31 Aug 2021. ## Builds on (most cited first, 30 of 30) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014) - [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016) - [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014) - [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015) - [Deep Audio-Visual Speech Recognition](https://mlsanity.com/paper/1809.02108/) (2018) - [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017) - [Lip Reading Sentences in the Wild](https://mlsanity.com/paper/1611.05358/) (2016) - [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020) - [Learning to Translate in Real-time with Neural Machine Translation](https://mlsanity.com/paper/1610.00388/) (2016) - [Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset](https://mlsanity.com/paper/2010.11395/) (2020) - [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018) - [SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation](https://mlsanity.com/paper/2011.02048/) (2020) - [LipNet: End-to-End Sentence-level Lipreading](https://mlsanity.com/paper/1611.01599/) (2016) - [Combining Residual Networks with LSTMs for Lipreading](https://mlsanity.com/paper/1703.04105/) (2017) - [Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecture](https://mlsanity.com/paper/1810.00108/) (2018) - [Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/1908.10992/) (2019) - [Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation](https://mlsanity.com/paper/1806.03661/) (2018) - [Towards Fast and Accurate Streaming End-to-End ASR](https://mlsanity.com/paper/2004.11544/) (2020) - [Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers](https://mlsanity.com/paper/1911.11502/) (2019) - [Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition](https://mlsanity.com/paper/2012.05481/) (2020) - [Lipreading with Long Short-Term Memory](https://mlsanity.com/paper/1601.08188/) (2016) - [Variational Memory Addressing in Generative Models](https://mlsanity.com/paper/1709.07116/) (2017) - [LCANet: End-to-End Lipreading with Cascaded Attention-CTC](https://mlsanity.com/paper/1803.04988/) (2018) - [Simultaneous Speech-to-Speech Translation System with Neural Incremental ASR, MT, and TTS](https://mlsanity.com/paper/2011.04845/) (2020) - [DualLip: A System for Joint Lip Reading and Generation](https://mlsanity.com/paper/2009.05784/) (2020) - [Exploring Pre-training with Alignments for RNN Transducer based End-to-End Speech Recognition](https://mlsanity.com/paper/2005.00572/) (2020) - [Learning to Generate with Memory](https://mlsanity.com/paper/1602.07416/) (2016) - [FastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire](https://mlsanity.com/paper/2008.02516/) (2020) - [Variational Memory Encoder-Decoder](https://mlsanity.com/paper/1807.09950/) (2018) ## Built on by (most cited first, 7 of 7) - [Gloss Attention for Gloss-free Sign Language Translation](https://mlsanity.com/paper/2307.07361/) (2023) - [TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation](https://mlsanity.com/paper/2205.12523/) (2022) - [MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition](https://mlsanity.com/paper/2303.05309/) (2023) - [TrainerAgent: Customizable and Efficient Model Training through LLM-Powered Multi-Agent System](https://mlsanity.com/paper/2311.06622/) (2023) - [DATE: Domain Adaptive Product Seeker for E-commerce](https://mlsanity.com/paper/2304.03669/) (2023) - [Language Model is a Branch Predictor for Simultaneous Machine Translation](https://mlsanity.com/paper/2312.14488/) (2023) - [Perception and Semantic Aware Regularization for Sequential Confidence Calibration](https://mlsanity.com/paper/2305.19498/) (2023) ## Abstract (arXiv) Lip reading, aiming to recognize spoken sentences according to the given video of lip movements without relying on the audio stream, has attracted great interest due to its application in many scenarios. Although prior works that explore lip reading have obtained salient achievements, they are all trained in a non-simultaneous manner where the predictions are generated requiring access to the full video. To breakthrough this constraint, we study the task of simultaneous lip reading and devise SimulLR, a simultaneous lip Reading transducer with attention-guided adaptive memory from three aspects: (1) To address the challenge of monotonic alignments while considering the syntactic structure of the generated sentences under simultaneous setting, we build a transducer-based model and design several effective training strategies including CTC pre-training, model warm-up and curriculum learning to promote the training of the lip reading transducer. (2) To learn better spatio-temporal representations for simultaneous encoder, we construct a truncated 3D convolution and time-restricted self-attention layer to perform the frame-to-frame interaction within a video segment containing fixed number of frames. (3) The history information is always limited due to the storage in real-time scenarios, especially for massive video data. Therefore, we devise a novel attention-guided adaptive memory to organize semantic information of history segments and enhance the visual representations with acceptable computation-aware latency. The experiments show that the SimulLR achieves the translation speedup 9.10× compared with the state-of-the-art non-simultaneous methods, and also obtains competitive results, which indicates the effectiveness of our proposed methods.