mlsanityData to 7 Oct 2026

mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition

arXiv 2502.01547 ↗First posted 3 Feb 2025eess.AS cs.CV cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

Audio-Visual Speech Recognition (AVSR) combines lip-based video with audio and can improve performance in noise, but most methods are trained only on English data. One limitation is the lack of large-scale multilingual video data, which makes it hard to train models from scratch. In this work, we propose mWhisper-Flamingo for multilingual AVSR which combines the strengths of a pre-trained audio model (Whisper) and video model (AV-HuBERT). To enable better multi-modal integration and improve the noisy multilingual performance, we introduce decoder modality dropout where the model is trained both on paired audio-visual inputs and separate audio/visual inputs. mWhisper-Flamingo achieves state-of-the-art WER on MuAViC, an AVSR dataset of 9 languages. Audio-visual mWhisper-Flamingo consistently outperforms audio-only Whisper on all languages in noisy conditions.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 42 of its 49 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

42 of 49 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  5. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  6. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  7. 2018Deep Audio-Visual Speech Recognitioncited by 318
  8. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  9. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  10. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  11. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  12. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
30 more
  1. 2020Lipreading using Temporal Convolutional Networkscited by 98
  2. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
  3. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  4. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  5. 2018End-to-end Audiovisual Speech Recognitioncited by 71
  6. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 71
  7. 2023ML-SUPERB: Multilingual Speech Universal PERformance Benchmarkcited by 70
  8. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 61
  9. 2020Discriminative Multi-modality Speech Recognitioncited by 56
  10. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  11. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 40
  12. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  13. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  14. 2024Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translationcited by 35
  15. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 30
  16. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 28
  17. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 24
  18. 2023Audio-Visual Efficient Conformer for Robust Speech Recognitioncited by 22
  19. 2023Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languagescited by 20
  20. 2024BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognitioncited by 20
  21. 2023Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledgecited by 16
  22. 2024Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputscited by 14
  23. 2023Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whispercited by 13
  24. 2024XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perceptioncited by 11
  25. 2024Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representationcited by 8
  26. 2024Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformercited by 7
  27. 2023Learning Cross-lingual Visual Speech Representationscited by 5
  28. 2023AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognitioncited by 5
  29. 2023Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Modelcited by 4
  30. 2024Tailored Design of Audio-Visual Speech Recognition Models using Branchformerscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Noise-Robust AV-ASR Using Visual Features Both in the Whisper Encoder and Decodercited by 2
  2. 2026When AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapsecited by 2
  3. 2025CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronizationcited by 1
  4. 2026VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognitioncited by 0