mlsanityData to 7 Oct 2026

Recurrent Neural Network Transducer for Audio-Visual Speech Recognition

arXiv 1911.04890 ↗First posted 8 Nov 2019eess.AS cs.CL cs.CVcited by 71

Not on the map. Why?

Abstract (arXiv)

This work presents a large-scale audio-visual speech recognition system based on a recurrent neural network transducer (RNN-T) architecture. To support the development of such a system, we built a large audio-visual (A/V) dataset of segmented utterances extracted from YouTube public videos, leading to 31k hours of audio-visual training content. The performance of an audio-only, visual-only, and audio-visual system are compared on two large-vocabulary test sets: a set of utterance segments from public YouTube videos called YTDEV18 and the publicly available LRS3-TED set. To highlight the contribution of the visual modality, we also evaluated the performance of our system on the YTDEV18 set artificially corrupted with background noise and overlapping speech. To the best of our knowledge, our system significantly improves the state-of-the-art on the LRS3-TED set.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 22; 2023: 23; 2024: 8; 2025: 9; 2026: 2.

71 papers.

Sort:

By area:

  1. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  2. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  3. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  4. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
  5. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  6. 2021Sub-word Level Lip Reading With Visual Attentioncited by 60
  7. 2021Deep Spoken Keyword Spotting: An Overviewcited by 43
  8. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  9. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
  10. 2021LiRA: Learning Visual Speech Representations from Audio through Self-supervisioncited by 41
  11. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  12. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
38 more
  1. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 29
  2. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 28
  3. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  4. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 24
  5. 2022Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognitioncited by 23
  6. 2023Audio-Visual Efficient Conformer for Robust Speech Recognitioncited by 22
  7. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  8. 2022Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learningcited by 21
  9. 2023SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervisioncited by 20
  10. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 20
  11. 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16
  12. 2023AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Modelcited by 16
  13. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 14
  14. 2022AVATAR: Unconstrained Audiovisual Speech Recognitioncited by 12
  15. 2024A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognitioncited by 12
  16. 2021Visual Keyword Spotting with Attentioncited by 11
  17. 2023Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mappingcited by 10
  18. 2025VALLR: Visual ASR Language Model for Lip Readingcited by 10
  19. 2023Conformers are All You Need for Visual Speech Recognitioncited by 9
  20. 2023Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognitioncited by 9
  21. 2025Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representationscited by 9
  22. 2023OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignmentcited by 8
  23. 2023Do VSR Models Generalize Beyond LRS3?cited by 8
  24. 2023Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognitioncited by 7
  25. 2023SparseVSR: Lightweight and Noise Robust Visual Speech Recognitioncited by 7
  26. 2023LiteVSR: Efficient Visual Speech Recognition by Learning from Speech Representations of Unlabeled Datacited by 6
  27. 2021Audio-Visual Speech Recognition is Worth 32×32×8 Voxelscited by 5
  28. 2022Relaxed Attention for Transformer Modelscited by 5
  29. 2023AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognitioncited by 5
  30. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
  31. 2022Best of Both Worlds: Multi-task Audio-Visual Automatic Speech Recognition and Active Speaker Detectioncited by 4
  32. 2023MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognitioncited by 4
  33. 2024Multi-Modal Retrieval For Large Language Model Based Speech Recognitioncited by 4
  34. 2025mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognitioncited by 4
  35. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  36. 2022Multi-task RNN-T with Semantic Decoder for Streamable Spoken Language Understandingcited by 3
  37. 2022Audio-visual multi-channel speech separation, dereverberation and recognitioncited by 3
  38. 2022End-to-End Multi-Person Audio/Visual Automatic Speech Recognitioncited by 3

50 most cited of 71 All 71 (JSON)