mlsanityData to 7 Oct 2026

MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation

arXiv 2303.00628 ↗First posted 1 Mar 2023cs.CL eess.AScited by 39

Not on the map. Why?

Abstract (arXiv)

We introduce MuAViC, a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languages. It is fully transcribed and covers 6 English-to-X translation as well as 6 X-to-English translation directions. To the best of our knowledge, this is the first open benchmark for audio-visual speech-to-text translation and the largest open benchmark for multilingual audio-visual speech recognition. Our baseline results show that MuAViC is effective for building noise-robust speech recognition and translation models. We make the corpus available at https://github.com/facebookresearch/muavic.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 21 of its 32 references that are in the map. First posted on arXiv 1 Mar 2023.

Builds on

21 of 32 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
  5. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  6. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  7. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  8. 2018Deep Audio-Visual Speech Recognitioncited by 318
  9. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  10. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  11. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
  12. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
9 more
  1. 2020In defence of metric learning for speaker recognitioncited by 123
  2. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  3. 2021The Multilingual TEDx Corpus for Speech Recognition and Translationcited by 101
  4. 2019ASR is all you need: cross-modal distillation for lip readingcited by 85
  5. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  6. 2020CoVoST: A Diverse Multilingual Speech-To-Text Translation Corpuscited by 57
  7. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  8. 2018Augmenting Librispeech with French Translations: A Multimodal Corpus for Direct Speech Translation Evaluationcited by 34
  9. 2019A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Readingcited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 11; 2025: 16; 2026: 10.

39 papers.

Sort:

By area:

  1. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 40
  2. 2024Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processingcited by 29
  3. 2024MultiTalk: Enhancing 3D Talking Head Generation Across Languages with Multilingual Video Datasetcited by 28
  4. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  5. 2024MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languagescited by 13
  6. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  7. 2024XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perceptioncited by 11
  8. 2025Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representationscited by 9
  9. 2025Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metricscited by 9
  10. 2024Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformercited by 7
  11. 2025Cocktail-Party Audio-Visual Speech Recognitioncited by 6
  12. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 6
27 more
  1. 2025mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognitioncited by 4
  2. 2025Large Multimodal Models for Low-Resource Languages: A Surveycited by 4
  3. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  4. 2025A Cocktail-Party Benchmark: Multi-Modal dataset and Comparative Evaluation Resultscited by 4
  5. 2024FFSTC: Fongbe to French Speech Translation Corpuscited by 3
  6. 2024ViSpeR: Multilingual Audio-Visual Speech Recognitioncited by 2
  7. 2024Tailored Design of Audio-Visual Speech Recognition Models using Branchformerscited by 2
  8. 2025CAM~OES: A Comprehensive Automatic Speech Recognition Benchmark for European Portuguesecited by 2
  9. 2026Noise-Robust AV-ASR Using Visual Features Both in the Whisper Encoder and Decodercited by 2
  10. 2024AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologiescited by 1
  11. 2025CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronizationcited by 1
  12. 2025ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognitioncited by 1
  13. 2025Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypothesescited by 1
  14. 2024Comparison of Conventional Hybrid and CTC/Attention Decoders for Continuous Visual Speech Recognitioncited by 0
  15. 2025Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditionscited by 0
  16. 2025MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translationcited by 0
  17. 2025Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generationcited by 0
  18. 2025OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusioncited by 0
  19. 2026Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASRcited by 0
  20. 2026Bootstrapping Audiovisual Speech Recognition in Zero-AV-Resource Scenarioscited by 0
  21. 2026AI-Driven Modular Services for Accessible Multilingual Education in Immersive Extended Reality Settings: Integrating Speech Processing, Translation, and Sign Language Renderingcited by 0
  22. 2026INTERACT: An AI-Driven Extended Reality Framework for Accesible Communication Featuring Real-Time Sign Language Interpretation and Emotion Recognitioncited by 0
  23. 2026FalAR: A Large-scale Speaker-Annotated European Portuguese Speech Corpus of Parliamentary Sessionscited by 0
  24. 2026Measuring User's Mental Models of Speech Translation in Human-AI Collaborationcited by 0
  25. 2026From Speech to Interaction: Analyzing Multimodal Systems in Cocktail-Party Scenarioscited by 0
  26. 2026Improving Audiovisual Speech Recognition through Synthetic Visual Data Augmentationcited by 0
  27. 2026Align Then Reason: A Multimodal Lip-Sync Judge for Dubbingcited by 0