mlsanityData to 14 Oct 2026

Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer

arXiv 2405.12983 ↗First posted 14 Mar 2024eess.AS cs.AI cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Humans are adept at leveraging visual cues from lip movements for recognizing speech in adverse listening conditions. Audio-Visual Speech Recognition (AVSR) models follow similar approach to achieve robust speech recognition in noisy conditions. In this work, we present a multilingual AVSR model incorporating several enhancements to improve performance and audio noise robustness. Notably, we adapt the recently proposed Fast Conformer model to process both audio and visual modalities using a novel hybrid CTC/RNN-T architecture. We increase the amount of audio-visual training data for six distinct languages, generating automatic transcriptions of unlabelled multilingual datasets (VoxCeleb2 and AVSpeech). Our proposed model achieves new state-of-the-art performance on the LRS3 dataset, reaching WER of 0.8%. On the recently introduced MuAViC benchmark, our model yields an absolute average-WER reduction of 11.9% in comparison to the original baseline. Finally, we demonstrate the ability of the proposed model to perform audio-only, visual-only, and audio-visual speech recognition at test time.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

31 of 40 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  6. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 525
  7. 2018Deep Audio-Visual Speech Recognitioncited by 322
  8. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  9. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  10. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  11. 2016Lip Reading Sentences in the Wildcited by 228
  12. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
19 more
  1. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 124
  2. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 114
  3. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 96
  4. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  5. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 75
  6. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 64
  7. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  8. 2019SpecAugment on Large Scale Datasetscited by 60
  9. 2020Discriminative Multi-modality Speech Recognitioncited by 58
  10. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 47
  11. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 39
  12. 2023MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translationcited by 39
  13. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 37
  14. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 31
  15. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 27
  16. 2023Audio-Visual Efficient Conformer for Robust Speech Recognitioncited by 22
  17. 2018Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognitioncited by 18
  18. 2023Conformers are All You Need for Visual Speech Recognitioncited by 9
  19. 2023On Robustness to Missing Video for Audiovisual Speech Recognitioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 1.

7 papers.

  1. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learnerscited by 42
  2. 2025Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representationscited by 9
  3. 2025mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognitioncited by 4
  4. 2024Tailored Design of Audio-Visual Speech Recognition Models using Branchformerscited by 2
  5. 2026When AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapsecited by 2
  6. 2025CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronizationcited by 1
  7. 2025A Study on Regularization-Based Continual Learning Methods for Indic ASRcited by 1