mlsanityData to 7 Oct 2026

AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations

arXiv 2302.06419 ↗First posted 10 Feb 2023eess.AS cs.AI cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

Self-supervision has shown great potential for audio-visual speech recognition by vastly reducing the amount of labeled data required to build good systems. However, existing methods are either not entirely end-to-end or do not train joint representations of both modalities. In this paper, we introduce AV-data2vec which addresses these challenges and builds audio-visual representations based on predicting contextualized representations which has been successful in the uni-modal case. The model uses a shared transformer encoder for both audio and video and can combine both modalities to improve speech recognition. Results on LRS3 show that AV-data2vec consistently outperforms existing methods under all settings with the same amount of data and model size.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 62 of its 80 references that are in the map. First posted on arXiv 10 Feb 2023.

Builds on

62 of 80 references.

References · 62

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  11. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  12. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
38 more
  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  6. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  8. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  9. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  10. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  11. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  12. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  13. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  14. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  15. 2018Deep Audio-Visual Speech Recognitioncited by 318
  16. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  17. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  18. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  19. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  20. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  21. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  22. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  23. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  24. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  25. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  26. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  27. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
  28. 2019ASR is all you need: cross-modal distillation for lip readingcited by 85
  29. 2019Generative Pre-Training for Speech with Autoregressive Predictive Codingcited by 82
  30. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  31. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  32. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  33. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 71
  34. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  35. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  36. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  37. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  38. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 61

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 12; 2025: 6; 2026: 3.

24 papers.

Sort:

By area:

  1. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  2. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 20
  3. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  4. 2024YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detectioncited by 16
  5. 2024Towards Hierarchical Spoken Language Dysfluency Modelingcited by 15
  6. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 14
  7. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  8. 2024XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perceptioncited by 11
  9. 2024Stutter-Solver: End-to-end Multi-lingual Dysfluency Detectioncited by 8
  10. 2024Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformercited by 7
  11. 2024Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognitioncited by 7
  12. 2023AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognitioncited by 5
12 more
  1. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
  2. 2025mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognitioncited by 4
  3. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  4. 2023Audio-visual fine-tuning of audio-only ASR modelscited by 3
  5. 2023Learning Separable Hidden Unit Contributions for Speaker-Adaptive Lip-Readingcited by 2
  6. 2024Speech Separation with Pretrained Frontend to Minimize Domain Mismatchcited by 1
  7. 2024PFML: Self-Supervised Learning of Time-Series Data Without Representation Collapsecited by 1
  8. 2025CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronizationcited by 1
  9. 2025Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusioncited by 1
  10. 2026Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognitioncited by 1
  11. 2026Synesthesia via Direct Latent Augmentation:Bypassing the Decode-Encode Loop for Cross-Modal Distillationcited by 0
  12. 2026When Speech Meets Lips: Interpretable Audio-Visual Synchronization for L2 Pronunciation Assessmentcited by 0