mlsanityData to 14 Oct 2026

AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognition

arXiv 2309.17395 ↗First posted 29 Sep 2023cs.LG cs.SD eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

Audio-visual speech contains synchronized audio and visual information that provides cross-modal supervision to learn representations for both automatic speech recognition (ASR) and visual speech recognition (VSR). We introduce continuous pseudo-labeling for audio-visual speech recognition (AV-CPL), a semi-supervised method to train an audio-visual speech recognition (AVSR) model on a combination of labeled and unlabeled videos with continuously regenerated pseudo-labels. Our models are trained for speech recognition from audio-visual inputs and can perform speech recognition using both audio and visual modalities, or only one modality. Our method uses the same audio-visual model for both supervised training and pseudo-label generation, mitigating the need for external speech recognition models to generate pseudo-labels. AV-CPL obtains significant improvements in VSR performance on the LRS3 dataset while maintaining practical ASR and AVSR performance. Finally, using visual-only speech data, our method is able to leverage unlabeled visual speech to improve VSR.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 50 of its 61 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

50 of 61 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  10. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  11. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  12. 2018Deep Audio-Visual Speech Recognitioncited by 322
38 more
  1. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  2. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  3. 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 187
  4. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  5. 2019Revisiting Self-Training for Neural Sequence Generationcited by 155
  6. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  7. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 151
  8. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 141
  9. 2019Self-Training for End-to-End Speech Recognitioncited by 129
  10. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  11. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  12. 2019ASR is all you need: cross-modal distillation for lip readingcited by 88
  13. 2020Iterative Pseudo-Labeling for Speech Recognitioncited by 83
  14. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  15. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 75
  16. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 74
  17. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 73
  18. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 68
  19. 2020Discriminative Multi-modality Speech Recognitioncited by 58
  20. 2018Large-Scale Visual Speech Recognitioncited by 57
  21. 2020SlimIPL: Language-Model-Free Iterative Pseudo-Labelingcited by 54
  22. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  23. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 47
  24. 2021LiRA: Learning Visual Speech Representations from Audio through Self-supervisioncited by 44
  25. 2021Momentum Pseudo-Labeling for Semi-Supervised Speech Recognitioncited by 40
  26. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 39
  27. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 37
  28. 2020Self-Training for End-to-End Speech Translationcited by 36
  29. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 31
  30. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 27
  31. 2022Flashlight: Enabling Innovation in Tools for Machine Learningcited by 25
  32. 2021Kaizen: Continuously improving teacher using Exponential Moving Average for semi-supervised speech recognitioncited by 21
  33. 2021Scaling ASR Improves Zero and Few Shot Learningcited by 16
  34. 2022Continuous Pseudo-Labeling from the Startcited by 11
  35. 2022Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillationcited by 9
  36. 2023Conformers are All You Need for Visual Speech Recognitioncited by 9
  37. 2022Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognitioncited by 5
  38. 2022Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Datacited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 15
  2. 2024dMel: Speech Tokenization made Simplecited by 11
  3. 2025mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognitioncited by 4
  4. 2025Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Modelscited by 4
  5. 2026Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognitioncited by 1