mlsanityData to 7 Oct 2026

Robust Self-Supervised Audio-Visual Speech Recognition

arXiv 2201.01763 ↗First posted 5 Jan 2022cs.SD cs.CV cs.LGcited by 72

Not on the map. Why?

Abstract (arXiv)

Audio-based automatic speech recognition (ASR) degrades significantly in noisy environments and is particularly vulnerable to interfering speech, as the model cannot determine which speaker to transcribe. Audio-visual speech recognition (AVSR) systems improve robustness by complementing the audio stream with the visual information that is invariant to noise and helps the model focus on the desired speaker. However, previous AVSR work focused solely on the supervised learning setup; hence the progress was hindered by the amount of labeled data available. In this work, we present a self-supervised AVSR framework built upon Audio-Visual HuBERT (AV-HuBERT), a state-of-the-art audio-visual speech representation learning model. On the largest available AVSR benchmark dataset LRS3, our approach outperforms prior state-of-the-art by ~50% (28.0% vs. 14.1%) using less than 10% of labeled data (433hr vs. 30hr) in the presence of babble noise, while reducing the WER of an audio-based model by over 75% (25.8% vs. 5.8%) on average.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 5 Jan 2022.

Builds on

18 of 26 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  5. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  6. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  7. 2018Deep Audio-Visual Speech Recognitioncited by 318
  8. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  9. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  10. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  11. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  12. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 71
6 more
  1. 2016Achieving Human Parity in Conversational Speech Recognitioncited by 63
  2. 2020Discriminative Multi-modality Speech Recognitioncited by 56
  3. 2020Single headed attention based sequence-to-sequence model for state-of-the-art results on Switchboardcited by 36
  4. 2021EasyCom: An Augmented Reality Dataset to Support Algorithms for Easy Communication in Noisy Environmentscited by 29
  5. 2020Improving noise robust automatic speech recognition with single-channel time-domain enhancement networkcited by 20
  6. 2020Super-Human Performance in Online Low-latency Recognition of Conversational Speechcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 25; 2024: 11; 2025: 17; 2026: 8.

72 papers.

Sort:

By area:

  1. 2022Masked Autoencoders that Listencited by 271
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  3. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  4. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 74
  5. 2022Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from Videoscited by 50
  6. 2023A-JEPA: Joint-Embedding Predictive Architecture Can Listencited by 50
  7. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  8. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  9. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  10. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  11. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  12. 2022Learning to Dub Movies via Hierarchical Prosody Modelscited by 28
38 more
  1. 2023Intelligible Lip-to-Speech Synthesis with Speech Unitscited by 28
  2. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 24
  3. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  4. 2022Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learningcited by 21
  5. 2023Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parsercited by 18
  6. 2024Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluationcited by 17
  7. 2023Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledgecited by 16
  8. 2023MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognitioncited by 14
  9. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  10. 2023Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mappingcited by 10
  11. 2023CAD -- Contextual Multi-modal Alignment for Dynamic AVQAcited by 10
  12. 2023Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognitioncited by 9
  13. 2022Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERTcited by 8
  14. 2023Do VSR Models Generalize Beyond LRS3?cited by 8
  15. 2023Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognitioncited by 7
  16. 2024Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extractioncited by 7
  17. 2024Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformercited by 7
  18. 2024Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognitioncited by 7
  19. 2025Cocktail-Party Audio-Visual Speech Recognitioncited by 6
  20. 2023AV2Wav: Diffusion-Based Re-synthesis from Continuous Self-supervised Features for Audio-Visual Speech Enhancementcited by 5
  21. 2023AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognitioncited by 5
  22. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
  23. 2022Visually-aware Acoustic Event Detection using Heterogeneous Graphscited by 4
  24. 2023MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognitioncited by 4
  25. 2023Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognitioncited by 4
  26. 2025mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognitioncited by 4
  27. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  28. 2025Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representationscited by 4
  29. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  30. 2023Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realismcited by 3
  31. 2023Audio-visual fine-tuning of audio-only ASR modelscited by 3
  32. 2025Visual Cues Support Robust Turn-taking Prediction in Noisecited by 3
  33. 2025Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Modelscited by 3
  34. 2023Optimizing Audio Augmentations for Contrastive Learning of Health-Related Acoustic Signalscited by 2
  35. 2023Learning Separable Hidden Unit Contributions for Speaker-Adaptive Lip-Readingcited by 2
  36. 2023AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Deepfake Detection of Frontal Face Videoscited by 2
  37. 2024Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?cited by 2
  38. 2024Transferable Adversarial Attacks against ASRcited by 2

50 most cited of 72 All 72 (JSON)