mlsanityData to 14 Oct 2026

Combining Residual Networks with LSTMs for Lipreading

arXiv 1703.04105 ↗First posted 12 Mar 2017cs.CVcited by 68

Not on the map. Why?

Abstract (arXiv)

We propose an end-to-end deep learning architecture for word-level visual speech recognition. The system is a combination of spatiotemporal convolutional, residual and bidirectional Long Short-Term Memory networks. We train and evaluate it on the Lipreading In-The-Wild benchmark, a challenging database of 500-size target-words consisting of 1.28sec video excerpts from BBC TV broadcasts. The proposed network attains word accuracy equal to 83.0, yielding 6.8 absolute improvement over the current state-of-the-art, without using information about word boundaries during training or testing.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 16; 2023: 18; 2024: 9; 2025: 9; 2026: 5.

68 papers.

Sort:

By area:

  1. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  2. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  3. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  4. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  5. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  6. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 68
  7. 2021Sub-word Level Lip Reading With Visual Attentioncited by 63
  8. 2022Lip to Speech Synthesis with Visual Context Attentional GANcited by 41
  9. 2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
  10. 2021Read and Attend: Temporal Localisation in Sign Language Videoscited by 34
  11. 2022Learning Long-Term Spatial-Temporal Graphs for Active Speaker Detectioncited by 30
  12. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 30
38 more
  1. 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
  2. 2022Training Strategies for Improved Lip-readingcited by 25
  3. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  4. 2023SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervisioncited by 21
  5. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 21
  6. 2022Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Videocited by 18
  7. 2022Speaker-adaptive Lip Reading with User-dependent Paddingcited by 17
  8. 2023AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Modelcited by 16
  9. 2023Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledgecited by 16
  10. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 15
  11. 2023Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognitioncited by 14
  12. 2022Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognitioncited by 12
  13. 2021Visual Keyword Spotting with Attentioncited by 11
  14. 2024XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perceptioncited by 11
  15. 2021LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Readingcited by 10
  16. 2022Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillationcited by 9
  17. 2023LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformerscited by 9
  18. 2022A Multimodal German Dataset for Automatic Lip Reading Systems and Transfer Learningcited by 8
  19. 2022Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERTcited by 8
  20. 2024Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representationcited by 8
  21. 2021SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memorycited by 7
  22. 2025Phoneme-Level Visual Speech Recognition via Point-Visual Fusion and Language Model Reconstructioncited by 7
  23. 2021Contrastive Learning of Global-Local Video Representationscited by 6
  24. 2022VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selectioncited by 6
  25. 2023AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translationcited by 6
  26. 2021LRWR: Large-Scale Benchmark for Lip Reading in Russian languagecited by 5
  27. 2022Is Lip Region-of-Interest Sufficient for Lipreading?cited by 5
  28. 2022Relaxed Attention for Transformer Modelscited by 5
  29. 2023Learning Cross-lingual Visual Speech Representationscited by 5
  30. 2023AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognitioncited by 5
  31. 2024Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Languagecited by 5
  32. 2021Advances and Challenges in Deep Lip Readingcited by 4
  33. 2023Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Videocited by 4
  34. 2024AlignVSR: Audio-Visual Cross-Modal Alignment for Visual Speech Recognitioncited by 4
  35. 2025LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognitioncited by 4
  36. 2022Learning Speaker-specific Lip-to-Speech Generationcited by 3
  37. 2023Large-scale unsupervised audio pre-training for video-to-speech synthesiscited by 3
  38. 2023Word-level Persian Lipreading Datasetcited by 2

50 most cited of 68 All 68 (JSON)