mlsanityData to 7 Oct 2026

Deep Audio-Visual Speech Recognition

arXiv 1809.02108 ↗First posted 6 Sep 2018cs.CVcited by 318

Not on the map. Why?

Abstract (arXiv)

The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem - unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.

Key facts. Cited by 318 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 34; 2022: 63; 2023: 87; 2024: 56; 2025: 57; 2026: 21.

318 papers.

Sort:

By area:

  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  4. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  5. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  6. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  7. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  8. 2022VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wildcited by 106
  9. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  10. 2023Emotional Speech-Driven Animation with Content-Emotion Disentanglementcited by 98
  11. 2023Identity-Preserving Talking Face Generation with Landmark and Appearance Priorscited by 97
  12. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
38 more
  1. 2023SyncTalk: The Devil is in the Synchronization for Talking Head Synthesiscited by 86
  2. 2023Self-Supervised Video Forensics by Audio-Visual Anomaly Detectioncited by 84
  3. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  4. 2024EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesiscited by 79
  5. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  6. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  7. 2022SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memorycited by 65
  8. 2022A General Survey on Attention Mechanisms in Deep Learningcited by 64
  9. 2021Sub-word Level Lip Reading With Visual Attentioncited by 60
  10. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learningcited by 53
  11. 2022Contrastive Positive Sample Propagation along the Audio-Visual Event Linecited by 50
  12. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  13. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  14. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
  15. 2021LiRA: Learning Visual Speech Representations from Audio through Self-supervisioncited by 41
  16. 2022Lip to Speech Synthesis with Visual Context Attentional GANcited by 41
  17. 2021Improving Multi-Modal Learning with Uni-Modal Teacherscited by 40
  18. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 40
  19. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  20. 2021Neural Dubber: Dubbing for Videos According to Scriptscited by 39
  21. 2021Audio-Visual Synchronisation in the wildcited by 39
  22. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  23. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  24. 2023STARSS23: An Audio-Visual Dataset of Spatial Recordings of Real Scenes with Spatiotemporal Annotations of Sound Eventscited by 39
  25. 2022An efficient encoder-decoder architecture with top-down attention for speech separationcited by 38
  26. 2022Temporal Sentence Grounding in Videos: A Survey and Future Directionscited by 37
  27. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  28. 2021The Right to Talk: An Audio-Visual Transformer Approachcited by 33
  29. 2024Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakescited by 32
  30. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  31. 2025SepPrune: Structured Pruning for Efficient Deep Speech Separationcited by 31
  32. 2022M^4I: Multi-modal Models Membership Inferencecited by 30
  33. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 28
  34. 2023Intelligible Lip-to-Speech Synthesis with Speech Unitscited by 28
  35. 2021Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separationcited by 27
  36. 2022Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localizationcited by 27
  37. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
  38. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27

50 most cited of 318 All 318 (JSON)