mlsanityData to 7 Oct 2026

XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception

arXiv 2403.14402 ↗First posted 21 Mar 2024cs.SD cs.CL eess.AScited by 11

Not on the map. Why?

Abstract (arXiv)

Speech recognition and translation systems perform poorly on noisy inputs, which are frequent in realistic environments. Augmenting these systems with visual signals has the potential to improve robustness to noise. However, audio-visual (AV) data is only available in limited amounts and for fewer languages than audio-only resources. To address this gap, we present XLAVS-R, a cross-lingual audio-visual speech representation model for noise-robust speech recognition and translation in over 100 languages. It is designed to maximize the benefits of limited multilingual AV pre-training data, by building on top of audio-only multilingual pre-training and simplifying existing pre-training schemes. Extensive evaluation on the MuAViC benchmark shows the strength of XLAVS-R on downstream audio-visual speech recognition and translation tasks, where it outperforms the previous state of the art by up to 18.5% WER and 4.7 BLEU given noisy AV inputs, and enables strong zero-shot audio-visual ability with audio-only fine-tuning.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

30 of 40 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  10. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  11. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 458
  12. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
18 more
  1. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  2. 2016Lip Reading Sentences in the Wildcited by 226
  3. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
  4. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  5. 2020Lipreading using Temporal Convolutional Networkscited by 98
  6. 2017Combining Residual Networks with LSTMs for Lipreadingcited by 66
  7. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  8. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  9. 2015Deep Multimodal Learning for Audio-Visual Speech Recognitioncited by 37
  10. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  11. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  12. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 24
  13. 2022Robust Data2vec: Noise-robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learningcited by 21
  14. 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16
  15. 2023MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognitioncited by 14
  16. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  17. 2022Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillationcited by 6
  18. 2023Audio-visual fine-tuning of audio-only ASR modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 1.

11 papers.

  1. 2024SpeechQE: Estimating the Quality of Direct Speech Translationcited by 10
  2. 2025Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representationscited by 9
  3. 2025mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognitioncited by 4
  4. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  5. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  6. 2024Tailored Design of Audio-Visual Speech Recognition Models using Branchformerscited by 2
  7. 2025AdaCS: Adaptive Normalization for Enhanced Code-Switching ASRcited by 2
  8. 2025CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronizationcited by 1
  9. 2025Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Modelscited by 1
  10. 2025Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypothesescited by 1
  11. 2026Measuring User's Mental Models of Speech Translation in Human-AI Collaborationcited by 0