mlsanityData to 14 Oct 2026

Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERT

arXiv 2205.07180 ↗First posted 15 May 2022eess.AS cs.CV cs.SDcited by 8

Not on the map. Why?

Abstract (arXiv)

This paper investigates self-supervised pre-training for audio-visual speaker representation learning where a visual stream showing the speaker's mouth area is used alongside speech as inputs. Our study focuses on the Audio-Visual Hidden Unit BERT (AV-HuBERT) approach, a recently developed general-purpose audio-visual speech pre-training framework. We conducted extensive experiments probing the effectiveness of pre-training and visual modality. Experimental results suggest that AV-HuBERT generalizes decently to speaker related downstream tasks, improving label efficiency by roughly ten fold for both audio-only and audio-visual speaker verification. We also show that incorporating visual information, even just the lip area, greatly improves the performance and noise robustness, reducing EER by 38% in the clean condition and 75% in noisy conditions.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 33 of its 44 references that are in the map. First posted on arXiv 15 May 2022.

Builds on

33 of 44 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2018Deep contextualized word representationscited by 2,290
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  9. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  12. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 628
21 more
  1. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  2. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  3. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  4. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  5. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  6. 2020In defence of metric learning for speaker recognitioncited by 126
  7. 2020Lipreading using Temporal Convolutional Networkscited by 102
  8. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 75
  9. 2017Speaker Diarization with LSTMcited by 73
  10. 2018VoiceFilter: Targeted Voice Separation by Speaker-Conditioned Spectrogram Maskingcited by 73
  11. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 68
  12. 2018Seeing Voices and Hearing Faces: Cross-modal biometric matchingcited by 60
  13. 2020Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learningcited by 33
  14. 2020Disentangled Speech Embeddings using Cross-modal Self-supervisioncited by 30
  15. 2020The IDLAB VoxSRC-20 Submission: Large Margin Fine-Tuning and Quality-Aware Score Calibration in DNN Based Speaker Verificationcited by 23
  16. 2021A Multi-View Approach To Audio-Visual Speaker Verificationcited by 23
  17. 2018Noise-tolerant Audio-visual Online Person Verification using an Attention-based Neural Network Fusioncited by 21
  18. 2021The SpeakIn System for VoxCeleb Speaker Recognition Challange 2021cited by 20
  19. 2020Semi-Supervised Contrastive Learning with Generalized Contrastive Loss and Its Application to Speaker Recognitioncited by 19
  20. 2020DurIAN-SC: Duration Informed Attention Network based Singing Voice Conversion Systemcited by 14
  21. 2021Self-supervised Speaker Recognition with Loss-gated Learningcited by 14

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 1; 2025: 3.

8 papers.

  1. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  2. 2023AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translationcited by 6
  3. 2023Cross-modal Audio-visual Co-learning for Text-independent Speaker Verificationcited by 3
  4. 2024Mechanisms of Multimodal Synchronization: Insights from Decoder-Based Video-Text-to-Speech Synthesiscited by 2
  5. 2023Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verificationcited by 1
  6. 2025DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibilitycited by 1
  7. 2025MuteSwap: Visual-informed Silent Video Identity Conversioncited by 0
  8. 2025Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voicecited by 0