mlsanityData to 7 Oct 2026

Lip Reading Sentences in the Wild

arXiv 1611.05358 ↗First posted 16 Nov 2016cs.CVcited by 226

Not on the map. Why?

Abstract (arXiv)

The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) a 'Watch, Listen, Attend and Spell' (WLAS) network that learns to transcribe videos of mouth motion to characters; (2) a curriculum learning strategy to accelerate training and to reduce overfitting; (3) a 'Lip Reading Sentences' (LRS) dataset for visual speech recognition, consisting of over 100,000 natural sentences from British television. The WLAS model trained on the LRS dataset surpasses the performance of all previous work on standard lip reading benchmark datasets, often by a significant margin. This lip reading performance beats a professional lip reader on videos from BBC television, and we also demonstrate that visual information helps to improve speech recognition performance even when the audio is available.

Key facts. Cited by 226 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 32; 2023: 61; 2024: 34; 2025: 41; 2026: 22.

226 papers.

Sort:

By area:

  1. 2021Improving Sign Language Translation with Monolingual Data by Sign Back-Translationcited by 177
  2. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  3. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  4. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
  5. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  6. 2022Localizing Visual Sounds the Easy Waycited by 81
  7. 2021KoDF: A Large-scale Korean DeepFake Detection Datasetcited by 80
  8. 2021Robust Audio-Visual Instance Discriminationcited by 80
  9. 2022DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Renderingcited by 80
  10. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  11. 2023SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Facescited by 67
  12. 2021Sub-word Level Lip Reading With Visual Attentioncited by 60
38 more
  1. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  2. 2023RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatarscited by 45
  3. 2021Deep Spoken Keyword Spotting: An Overviewcited by 43
  4. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  5. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
  6. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 41
  7. 2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
  8. 2021Audio-Visual Synchronisation in the wildcited by 39
  9. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  10. 2023A Light Weight Model for Active Speaker Detectioncited by 39
  11. 2024Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Modelcited by 37
  12. 2021MAAS: Multi-modal Assignation for Active Speaker Detectioncited by 35
  13. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  14. 2023MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentionscited by 29
  15. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 29
  16. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 28
  17. 2022Learning to Dub Movies via Hierarchical Prosody Modelscited by 28
  18. 2023Speech Driven Video Editing via an Audio-Conditioned Diffusion Modelcited by 28
  19. 2023Intelligible Lip-to-Speech Synthesis with Speech Unitscited by 28
  20. 2021Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separationcited by 27
  21. 2023LaughTalk: Expressive 3D Talking Head Generation with Laughtercited by 27
  22. 2024Audio-Visual Segmentation via Unlabeled Frame Exploitationcited by 27
  23. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  24. 2023LoCoNet: Long-Short Context Network for Active Speaker Detectioncited by 24
  25. 2021Can audio-visual integration strengthen robustness under multimodal attacks?cited by 23
  26. 2021V2C: Visual Voice Cloningcited by 23
  27. 2022Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognitioncited by 23
  28. 2022Training Strategies for Improved Lip-readingcited by 23
  29. 2024Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learningcited by 23
  30. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  31. 2023Audio-Visual Efficient Conformer for Robust Speech Recognitioncited by 22
  32. 2023DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embeddingcited by 22
  33. 2021Look Who's Talking: Active Speaker Detection in the Wildcited by 21
  34. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  35. 2023Lip-to-Speech Synthesis in the Wild with Multi-task Learningcited by 21
  36. 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
  37. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  38. 2022Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videoscited by 18

50 most cited of 226 All 226 (JSON)