mlsanityData to 7 Oct 2026

u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality

arXiv 2207.07036 ↗First posted 14 Jul 2022cs.CL cs.AI cs.CVcited by 36

Not on the map. Why?

Abstract (arXiv)

While audio-visual speech models can yield superior performance and robustness compared to audio-only models, their development and adoption are hindered by the lack of labeled and unlabeled audio-visual data and the cost to deploy one model per modality. In this paper, we present u-HuBERT, a self-supervised pre-training framework that can leverage both multimodal and unimodal speech with a unified masked cluster prediction objective. By utilizing modality dropout during pre-training, we demonstrate that a single fine-tuned model can achieve performance on par or better than the state-of-the-art modality-specific models. Moreover, our model fine-tuned only on audio can perform well with audio-visual and visual speech input, achieving zero-shot modality generalization for multiple speech processing tasks. In particular, our single model yields 1.2%/1.4%/27.2% speech recognition word error rate on LRS3 with audio-visual/audio/visual input. Codes and models are available at https://github.com/facebookresearch/av_hubert

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 50 of its 69 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

50 of 69 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020Language Models are Few-Shot Learnerscited by 32,199
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  12. 2019Cross-lingual Language Model Pretrainingcited by 1,088
38 more
  1. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  2. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  3. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  4. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  6. 2019How multilingual is Multilingual BERT?cited by 642
  7. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  8. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  9. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  10. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  11. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  12. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  13. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  14. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  15. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  16. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  17. 2021MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglementcited by 189
  18. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  19. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
  20. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  21. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  22. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  23. 2020Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameterscited by 110
  24. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
  25. 2019ASR is all you need: cross-modal distillation for lip readingcited by 85
  26. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  27. 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 72
  28. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  29. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 71
  30. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  31. 2021Sub-word Level Lip Reading With Visual Attentioncited by 60
  32. 2020Discriminative Multi-modality Speech Recognitioncited by 56
  33. 2020Learning Robust and Multilingual Speech Representationscited by 39
  34. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34
  35. 2020Far-Field Automatic Speech Recognitioncited by 32
  36. 2021A Multi-View Approach To Audio-Visual Speaker Verificationcited by 23
  37. 2021Multilingual Unsupervised Neural Machine Translation with Denoising Adapterscited by 22
  38. 2019EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networkscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 12; 2024: 10; 2025: 11; 2026: 2.

36 papers.

Sort:

By area:

  1. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  2. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  3. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  4. 2024Foundation Models for Music: A Surveycited by 34
  5. 2023Toward Joint Language Modeling for Speech Units and Textcited by 27
  6. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 24
  7. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  8. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 20
  9. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  10. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 14
  11. 2024XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perceptioncited by 11
  12. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
24 more
  1. 2023LANISTR: Multimodal Learning from Structured and Unstructured Datacited by 10
  2. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  3. 2023Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognitioncited by 9
  4. 2025Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representationscited by 9
  5. 2025From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speechcited by 9
  6. 2022ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancementcited by 8
  7. 2023Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognitioncited by 7
  8. 2024Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformercited by 7
  9. 2024Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognitioncited by 7
  10. 2024Multimodal Emotion Recognition with Vision-language Prompting and Modality Dropoutcited by 7
  11. 2023AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognitioncited by 5
  12. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
  13. 2023MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognitioncited by 4
  14. 2025mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognitioncited by 4
  15. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  16. 2023Audio-visual fine-tuning of audio-only ASR modelscited by 3
  17. 2025Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Modelscited by 3
  18. 2023Spatial HuBERT: Self-supervised Spatial Speech Representation Learning for a Single Talker from Multi-channel Audiocited by 2
  19. 2024Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognitioncited by 1
  20. 2025Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deploymentcited by 1
  21. 2025CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronizationcited by 1
  22. 2025Towards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audiocited by 1
  23. 2026Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognitioncited by 1
  24. 2026TSMD: Temporal-Stream Modality Dropout for Robust Video Highlight Detectioncited by 0