mlsanityData to 7 Oct 2026

Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring

arXiv 2303.08536 ↗First posted 15 Mar 2023cs.MM cs.CV cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

This paper deals with Audio-Visual Speech Recognition (AVSR) under multimodal input corruption situations where audio inputs and visual inputs are both corrupted, which is not well addressed in previous research directions. Previous studies have focused on how to complement the corrupted audio inputs with the clean visual inputs with the assumption of the availability of clean visual inputs. However, in real life, clean visual inputs are not always accessible and can even be corrupted by occluded lip regions or noises. Thus, we firstly analyze that the previous AVSR models are not indeed robust to the corruption of multimodal input streams, the audio and the visual inputs, compared to uni-modal models. Then, we design multimodal input corruption modeling to develop robust AVSR models. Lastly, we propose a novel AVSR framework, namely Audio-Visual Reliability Scoring module (AV-RelScore), that is robust to the corrupted multimodal inputs. The AV-RelScore can determine which input modal stream is reliable or not for the prediction and also can exploit the more reliable streams in prediction. The effectiveness of the proposed method is evaluated with comprehensive experiments on popular benchmark databases, LRS2 and LRS3. We also show that the reliability scores obtained by AV-RelScore well reflect the degree of corruption and make the proposed model focus on the reliable multimodal representations.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 46 of its 77 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

46 of 77 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  7. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  8. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  9. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  10. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  11. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  12. 2018Deep Audio-Visual Speech Recognitioncited by 318
34 more
  1. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  2. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  3. 2016Lip Reading Sentences in the Wildcited by 226
  4. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  5. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  6. 2014Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognitioncited by 111
  7. 2020Lipreading using Temporal Convolutional Networkscited by 98
  8. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 94
  9. 2016LipNet: End-to-End Sentence-level Lipreadingcited by 79
  10. 2018End-to-end Audiovisual Speech Recognitioncited by 71
  11. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 71
  12. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 61
  13. 2021Sub-word Level Lip Reading With Visual Attentioncited by 60
  14. 2020Discriminative Multi-modality Speech Recognitioncited by 56
  15. 2020Robust Object Detection under Occlusion with Context-Aware CompositionalNetscited by 54
  16. 2022Lip to Speech Synthesis with Visual Context Attentional GANcited by 41
  17. 2019Hearing Lips: Improving Lip Reading by Distilling Speech Recognizerscited by 39
  18. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  19. 2020Audio-visual Recognition of Overlapped speech for the LRS2 datasetcited by 38
  20. 2015Deep Multimodal Learning for Audio-Visual Speech Recognitioncited by 37
  21. 2021Towards efficient models for real-time deep noise suppressioncited by 28
  22. 2020Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognitioncited by 27
  23. 2022Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognitioncited by 23
  24. 2020Deformation Flow Based Two-Stream Network for Lip Readingcited by 22
  25. 2020Deep Feature Augmentation for Occluded Image Classificationcited by 21
  26. 2023Lip-to-Speech Synthesis in the Wild with Multi-task Learningcited by 21
  27. 2020Mutual Information Maximization for Effective Lip Readingcited by 18
  28. 2022Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Videocited by 18
  29. 2022Speaker-adaptive Lip Reading with User-dependent Paddingcited by 17
  30. 2023Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognitioncited by 13
  31. 2013Improvements to deep convolutional neural networks for LVCSRcited by 11
  32. 2022Delving into High-Quality Synthetic Face Occlusion Segmentation Datasetscited by 11
  33. 2019Multi-Grained Spatio-temporal Modeling for Lip-readingcited by 10
  34. 2022VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selectioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 7; 2025: 20; 2026: 10.

42 papers.

Sort:

By area:

  1. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 40
  2. 2024Towards Open-Vocabulary Audio-Visual Event Localizationcited by 26
  3. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  4. 2023Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledgecited by 16
  5. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  6. 2024A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognitioncited by 12
  7. 2023Textless Unit-to-Unit training for Many-to-Many Multilingual Speech-to-Speech Translationcited by 11
  8. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  9. 2025Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representationscited by 9
  10. 2024MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognitioncited by 8
  11. 2025AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generationcited by 6
  12. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
30 more
  1. 2025AV-Dialog: Spoken Dialogue Models with Audio-Visual Inputcited by 5
  2. 2023Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokenscited by 4
  3. 2024TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languagescited by 4
  4. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  5. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  6. 2025Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Modelscited by 3
  7. 2026The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?cited by 3
  8. 2024Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?cited by 2
  9. 2025Contextual Speech Extraction: Leveraging Textual History as an Implicit Cue for Target Speech Extractioncited by 2
  10. 2025textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Timecited by 2
  11. 2025GLip: A Global-Local Integrated Progressive Framework for Robust Visual Speech Recognitioncited by 2
  12. 2026When AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapsecited by 2
  13. 2024LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognitioncited by 1
  14. 2025Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deploymentcited by 1
  15. 2025Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extractioncited by 1
  16. 2025MLLM-based Speech Recognition: When and How is Multimodality Beneficial?cited by 1
  17. 2025AD-AVSR: Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognitioncited by 1
  18. 2025Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusioncited by 1
  19. 2025Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypothesescited by 1
  20. 2026Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognitioncited by 1
  21. 2023Fusion for Visual-Infrared Person ReID in Real-World Surveillance Using Corrupted Multimodal Datacited by 0
  22. 2025ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extractioncited by 0
  23. 2025RobustA: Robust Anomaly Detection in Multimodal Datacited by 0
  24. 2026Purification Before Fusion: Toward Mask-Free Speech Enhancement for Robust Audio-Visual Speech Recognitioncited by 0
  25. 2026Layer-Specific Lipschitz Modulation for Fault-Tolerant Multimodal Representation Learningcited by 0
  26. 2026Cross-Modal Masking for Robust Silent Speech Synthesis Using sEMG and Lipreadingcited by 0
  27. 2026VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognitioncited by 0
  28. 2026DoubleHelix: Structured Cross-Modal Fusion for Audio-Visual Speech Recognition with LLMscited by 0
  29. 2026AVSRBench: A Multi-Condition AVSR Benchmarkcited by 0
  30. 2026OP-CAD: On-Policy Clean-Audio Distillation for Robust Audio-Visual Reasoningcited by 0