mlsanityData to 7 Oct 2026

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

arXiv 2201.02184 ↗First posted 5 Jan 2022eess.AS cs.CV cs.SDcited by 277

Not on the map. Why?

Abstract (arXiv)

Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker's lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former state-of-the-art approach (33.6%) trained with a thousand times more transcribed video data (31K hours). The lip-reading WER is further reduced to 26.9% when using all 433 hours of labeled data from LRS3 and combined with self-training. Using our audio-visual representation on the same benchmark for audio-only speech recognition leads to a 40% relative WER reduction over the state-of-the-art performance (1.3% vs 2.3%). Our code and models are available at https://github.com/facebookresearch/av_hubert

Key facts. Cited by 277 papers in the map (data through 7 Oct 2026). Builds on 40 of its 52 references that are in the map. First posted on arXiv 5 Jan 2022.

Builds on

40 of 52 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2016Layer Normalizationcited by 5,579
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  9. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  10. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  11. 2017Look, Listen and Learncited by 452
  12. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
28 more
  1. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  2. 2018Deep Audio-Visual Speech Recognitioncited by 318
  3. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  4. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  5. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  6. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  7. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  8. 2018Lip Movements Generation at a Glancecited by 154
  9. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  10. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  11. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  12. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  13. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  14. 2020BSL-1K: Scaling up co-articulated sign language recognition using mouthing cuescited by 106
  15. 2020Lipreading using Temporal Convolutional Networkscited by 98
  16. 2019ASR is all you need: cross-modal distillation for lip readingcited by 85
  17. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  18. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 71
  19. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 66
  20. 2020Data Augmenting Contrastive Learning of Speech Representations in the Time Domaincited by 63
  21. 2018Large-Scale Visual Speech Recognitioncited by 57
  22. 2020Discriminative Multi-modality Speech Recognitioncited by 56
  23. 2020Parameter Efficient Multimodal Transformers for Video Representation Learningcited by 52
  24. 2021LiRA: Learning Visual Speech Representations from Audio through Self-supervisioncited by 41
  25. 2020Modality Dropout for Improved Performance-driven Talking Facescited by 32
  26. 2020Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervisioncited by 19
  27. 2021Multi-Modal Pre-Training for Automated Speech Recognitioncited by 10
  28. 2019Evolving Losses for Unlabeled Video Representation Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 28; 2023: 62; 2024: 69; 2025: 74; 2026: 44.

277 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  3. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  4. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 155
  5. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  6. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  7. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 108
  8. 2023Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expertcited by 98
  9. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  10. 2022Open-Domain Sign Language Translation Learned from Online Videocited by 72
  11. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  12. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
38 more
  1. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  2. 2022MAViL: Masked Audio-Video Learnerscited by 56
  3. 2022Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from Videoscited by 50
  4. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  5. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  6. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 40
  7. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  8. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  9. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 39
  10. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  11. 2022SVTS: Scalable Video-to-Speech Synthesiscited by 35
  12. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  13. 2024Foundation Models for Music: A Surveycited by 34
  14. 2024HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognitioncited by 30
  15. 2022iQuery: Instruments as Queries for Audio-Visual Sound Separationcited by 29
  16. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 29
  17. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 28
  18. 2022Learning to Dub Movies via Hierarchical Prosody Modelscited by 28
  19. 2023Intelligible Lip-to-Speech Synthesis with Speech Unitscited by 28
  20. 2024Masked Image Modeling: A Surveycited by 28
  21. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  22. 2024A review of deep learning-based information fusion techniques for multimodal medical image classificationcited by 25
  23. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 24
  24. 2025Multi-modal Time Series Analysis: A Tutorial and Surveycited by 24
  25. 2024FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matchingcited by 23
  26. 2024Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learningcited by 23
  27. 2022TVLT: Textless Vision-Language Transformercited by 22
  28. 2023DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embeddingcited by 22
  29. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  30. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  31. 2022Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learningcited by 21
  32. 2023SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervisioncited by 20
  33. 2024FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audiocited by 20
  34. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 20
  35. 2023AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Modelscited by 19
  36. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  37. 2023Emotional Listener Portrait: Neural Listener Head Generation with Emotioncited by 17
  38. 2023Detecting Deepfakes Without Seeing Anycited by 17

50 most cited of 277 All 277 (JSON)