Spatio-Temporal Attention Mechanism and Knowledge Distillation for Lip Reading
On the map: Knowledge distillation → Speech and audio distillation
Abstract (arXiv)
Despite the advancement in the domain of audio and audio-visual speech recognition, visual speech recognition systems are still quite under-explored due to the visual ambiguity of some phonemes. In this work, we propose a new lip-reading model that combines three contributions. First, the model front-end adopts a spatio-temporal attention mechanism to help extract the informative data from the input visual frames. Second, the model back-end utilizes a sequence-level and frame-level Knowledge Distillation (KD) techniques that allow leveraging audio data during the visual model training. Third, a data preprocessing pipeline is adopted that includes facial landmarks detection-based lip-alignment. On LRW lip-reading dataset benchmark, a noticeable accuracy improvement is demonstrated; the spatio-temporal attention, Knowledge Distillation, and lip-alignment contributions achieved 88.43%, 88.64%, and 88.37% respectively.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 23 of its 36 references that are in the map. First posted on arXiv 7 Aug 2021.
Builds on
23 of 36 references.
References · 23
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,882
- 2018Deep Audio-Visual Speech Recognitioncited by 318
- 2016Lip Reading Sentences in the Wildcited by 226
- 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
- 2020Lipreading using Temporal Convolutional Networkscited by 98
- 2019ASR is all you need: cross-modal distillation for lip readingcited by 85
- 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
- 2020Discriminative Multi-modality Speech Recognitioncited by 56
- 2018LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wildcited by 51
- 2020Towards Practical Lipreading with Distilled and Efficient Modelscited by 44
- 2019Hearing Lips: Improving Lip Reading by Distilling Speech Recognizerscited by 39
- 2020Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognitioncited by 27
11 more
- 2018Deep Lip Reading: a comparison of models and an online applicationcited by 26
- 2019Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreadingcited by 22
- 2020Deformation Flow Based Two-Stream Network for Lip Readingcited by 22
- 2020Learn an Effective Lip Reading Model without Painscited by 19
- 2020Mutual Information Maximization for Effective Lip Readingcited by 18
- 2020Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Readingcited by 12
- 2019Multi-Grained Spatio-temporal Modeling for Lip-readingcited by 10
- 20173D Convolutional Neural Networks for Cross Audio-Visual Matching Recognitioncited by 9
- 2020Synchronous Bidirectional Learning for Multilingual Lip Readingcited by 6
- 2020Lip-reading with Hierarchical Pyramidal Convolution and Self-Attentioncited by 3
- 2018Video-based Person Re-identification Using Spatial-Temporal Attention Networkscited by 1
Built on by
Citing papers per year: 2023: 1.
1 paper.