mlsanityData to 14 Oct 2026

Advances and Challenges in Deep Lip Reading

arXiv 2110.07879 ↗First posted 15 Oct 2021cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Driven by deep learning techniques and large-scale datasets, recent years have witnessed a paradigm shift in automatic lip reading. While the main thrust of Visual Speech Recognition (VSR) was improving accuracy of Audio Speech Recognition systems, other potential applications, such as biometric identification, and the promised gains of VSR systems, have motivated extensive efforts on developing the lip reading technology. This paper provides a comprehensive survey of the state-of-the-art deep learning based VSR research with a focus on data challenges, task-specific complications, and the corresponding solutions. Advancements in these directions will expedite the transformation of silent speech interface from theory to practice. We also discuss the main modules of a VSR pipeline and the influential datasets. Finally, we introduce some typical VSR application concerns and impediments to real-world scenarios as well as future research directions.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 70 of its 143 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

70 of 143 references.

References · 70

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  5. 2018SlowFast Networks for Video Recognitioncited by 1,867
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  7. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
  8. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  9. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  10. 2017One pixel attack for fooling deep neural networkscited by 582
  11. 2018Born Again Neural Networkscited by 546
  12. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
38 more
  1. 2019Object Detection in 20 Years: A Surveycited by 531
  2. 2018Deep Learning for Generic Object Detection: A Surveycited by 392
  3. 2018Deep Audio-Visual Speech Recognitioncited by 322
  4. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  5. 20193D Morphable Face Models -- Past, Present and Futurecited by 270
  6. 2015Highway Networkscited by 260
  7. 2016Lip Reading Sentences in the Wildcited by 228
  8. 2017Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?cited by 221
  9. 2019RetinaFace: Single-stage Dense Face Localisation in the Wildcited by 165
  10. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 162
  11. 2016Improving the Robustness of Deep Neural Networks via Stability Trainingcited by 156
  12. 2015Character-Aware Neural Language Modelscited by 132
  13. 2014Return of the Devil in the Details: Delving Deep into Convolutional Netscited by 122
  14. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 111
  15. 2020Lipreading using Temporal Convolutional Networkscited by 102
  16. 2019ASR is all you need: cross-modal distillation for lip readingcited by 88
  17. 2020Understanding and Improving Knowledge Distillationcited by 83
  18. 2016LipNet: End-to-End Sentence-level Lipreadingcited by 80
  19. 2018End-to-end Audiovisual Speech Recognitioncited by 73
  20. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 68
  21. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 64
  22. 2020Discriminative Multi-modality Speech Recognitioncited by 58
  23. 2018Large-Scale Visual Speech Recognitioncited by 57
  24. 2018LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wildcited by 51
  25. 2020Towards Practical Lipreading with Distilled and Efficient Modelscited by 45
  26. 2019Hearing Lips: Improving Lip Reading by Distilling Speech Recognizerscited by 41
  27. 2015Deep Multimodal Learning for Audio-Visual Speech Recognitioncited by 38
  28. 2020Audio-visual Recognition of Overlapped speech for the LRS2 datasetcited by 38
  29. 2020Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognitioncited by 30
  30. 2018Deep Lip Reading: a comparison of models and an online applicationcited by 26
  31. 2020Lip-reading with Densely Connected Temporal Convolutional Networkscited by 26
  32. 2019A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Readingcited by 25
  33. 2019Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreadingcited by 22
  34. 2020Deformation Flow Based Two-Stream Network for Lip Readingcited by 22
  35. 2020Learn an Effective Lip Reading Model without Painscited by 19
  36. 2020Mutual Information Maximization for Effective Lip Readingcited by 18
  37. 2018LCANet: End-to-End Lipreading with Cascaded Attention-CTCcited by 15
  38. 2017Towards Estimating the Upper Bound of Visual-Speech Recognition: The Visual Lip-Reading Feasibility Databasecited by 13

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 1.

4 papers.

  1. 2023Learning Separable Hidden Unit Contributions for Speaker-Adaptive Lip-Readingcited by 2
  2. 2023End-to-End Lip Reading in Romanian with Cross-Lingual Domain Adaptation and Lateral Inhibitioncited by 1
  3. 2024JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognitioncited by 0
  4. 2025Integrating Persian Lip Reading in Surena-V Humanoid Robot for Human-Robot Interactioncited by 0