mlsanityData to 7 Oct 2026

BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition

arXiv 2404.02098 ↗First posted 2 Apr 2024cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

Self-supervision has recently shown great promise for learning visual and auditory speech representations from unlabelled data. In this work, we propose BRAVEn, an extension to the recent RAVEn method, which learns speech representations entirely from raw audio-visual data. Our modifications to RAVEn enable BRAVEn to achieve state-of-the-art results among self-supervised methods in various settings. Moreover, we observe favourable scaling behaviour by increasing the amount of unlabelled data well beyond other self-supervised works. In particular, we achieve 20.0% / 1.7% word error rate for VSR / ASR on the LRS3 test set, with only 30 hours of labelled data and no external ASR models. Our results suggest that readily available unlabelled audio-visual data can largely replace costly transcribed data.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 28 of its 31 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

28 of 31 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2016Layer Normalizationcited by 5,579
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  8. 2016Deep Networks with Stochastic Depthcited by 973
  9. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  11. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  12. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
16 more
  1. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  2. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  3. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  4. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
  5. 2019ASR is all you need: cross-modal distillation for lip readingcited by 85
  6. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 71
  7. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 66
  8. 2021Sub-word Level Lip Reading With Visual Attentioncited by 60
  9. 2018Large-Scale Visual Speech Recognitioncited by 57
  10. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  11. 2021LiRA: Learning Visual Speech Representations from Audio through Self-supervisioncited by 41
  12. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  13. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  14. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 28
  15. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 24
  16. 2023SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervisioncited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 4.

20 papers.

Sort:

By area:

  1. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  2. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  3. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 14
  4. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  5. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  6. 2025Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representationscited by 9
  7. 2024Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognitioncited by 7
  8. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
  9. 2025mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognitioncited by 4
  10. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  11. 2025LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Modelscited by 2
  12. 2025Towards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audiocited by 1
8 more
  1. 2025Diffusion-Based Unsupervised Audio-Visual Speech Separation in Noisy Environments with Noise Priorcited by 1
  2. 2025Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypothesescited by 1
  3. 2025Investigating self-supervised representations for audio-visual deepfake detectioncited by 1
  4. 2026Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognitioncited by 1
  5. 2025Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditionscited by 0
  6. 2026SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Samplingcited by 0
  7. 2026From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarioscited by 0
  8. 2026Foundation Models are Implicit Deepfake Detectorscited by 0