mlsanityData to 14 Oct 2026

Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition

arXiv 2207.06020 ↗First posted 13 Jul 2022cs.SD cs.AI cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

This paper focuses on designing a noise-robust end-to-end Audio-Visual Speech Recognition (AVSR) system. To this end, we propose Visual Context-driven Audio Feature Enhancement module (V-CAFE) to enhance the input noisy audio speech with a help of audio-visual correspondence. The proposed V-CAFE is designed to capture the transition of lip movements, namely visual context and to generate a noise reduction mask by considering the obtained visual context. Through context-dependent modeling, the ambiguity in viseme-to-phoneme mapping can be refined for mask generation. The noisy representations are masked out with the noise reduction mask resulting in enhanced audio features. The enhanced audio features are fused with the visual features and taken to an encoder-decoder model composed of Conformer and Transformer for speech recognition. We show the proposed end-to-end AVSR with the V-CAFE can further improve the noise-robustness of AVSR. The effectiveness of the proposed method is evaluated in noisy speech recognition and overlapped speech recognition experiments using the two largest audio-visual datasets, LRS2 and LRS3.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 13 Jul 2022.

Builds on

23 of 36 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2018Deep Audio-Visual Speech Recognitioncited by 322
  8. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  9. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  10. 2016Lip Reading Sentences in the Wildcited by 228
  11. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  12. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 74
11 more
  1. 2018End-to-end Audiovisual Speech Recognitioncited by 73
  2. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 64
  3. 2020Discriminative Multi-modality Speech Recognitioncited by 58
  4. 2019Hearing Lips: Improving Lip Reading by Distilling Speech Recognizerscited by 41
  5. 2022Lip to Speech Synthesis with Visual Context Attentional GANcited by 41
  6. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  7. 2015Deep Multimodal Learning for Audio-Visual Speech Recognitioncited by 38
  8. 2020Audio-visual Recognition of Overlapped speech for the LRS2 datasetcited by 38
  9. 2022Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Videocited by 31
  10. 2022Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Videocited by 18
  11. 2020Visual Speech Enhancement Without A Real Visual Streamcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 2; 2025: 10; 2026: 4.

24 papers.

Sort:

By area:

  1. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
  2. 2023AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Modelcited by 16
  3. 2023Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledgecited by 16
  4. 2023Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech Recognitioncited by 14
  5. 2023AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representationcited by 14
  6. 2024A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognitioncited by 12
  7. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  8. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  9. 2025Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representationscited by 9
  10. 2023Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognitioncited by 7
  11. 2024Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognitioncited by 7
  12. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
12 more
  1. 2023Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Videocited by 4
  2. 2023Exploring Phonetic Context-Aware Lip-Sync For Talking Face Generationcited by 4
  3. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  4. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  5. 2025Contextual Speech Extraction: Leveraging Textual History as an Implicit Cue for Target Speech Extractioncited by 2
  6. 2025Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deploymentcited by 1
  7. 2025Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Modelscited by 1
  8. 2025Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypothesescited by 1
  9. 2026Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognitioncited by 1
  10. 2026Purification Before Fusion: Toward Mask-Free Speech Enhancement for Robust Audio-Visual Speech Recognitioncited by 0
  11. 2026Materialistic RIR: Material Conditioned Realistic RIR Generationcited by 0
  12. 2026Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognitioncited by 0