mlsanityData to 7 Oct 2026

Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition

arXiv 2305.09212 ↗First posted 16 May 2023eess.AS cs.CV cs.MMcited by 9

Not on the map. Why?

Abstract (arXiv)

Audio-visual speech recognition (AVSR) research has gained a great success recently by improving the noise-robustness of audio-only automatic speech recognition (ASR) with noise-invariant visual information. However, most existing AVSR approaches simply fuse the audio and visual features by concatenation, without explicit interactions to capture the deep correlations between them, which results in sub-optimal multimodal representations for downstream speech recognition task. In this paper, we propose a cross-modal global interaction and local alignment (GILA) approach for AVSR, which captures the deep audio-visual (A-V) correlations from both global and local perspectives. Specifically, we design a global interaction model to capture the A-V complementary relationship on modality level, as well as a local alignment approach to model the A-V temporal consistency on frame level. Such a holistic view of cross-modal correlations enable better multimodal representations for AVSR. Experiments on public benchmarks LRS3 and LRS2 show that our GILA outperforms the supervised learning state-of-the-art.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 45 of its 76 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

45 of 76 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Layer Normalizationcited by 5,579
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  10. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  11. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  12. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
33 more
  1. 2018Deep Audio-Visual Speech Recognitioncited by 318
  2. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  3. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  4. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  5. 2016Lip Reading Sentences in the Wildcited by 226
  6. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  7. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  8. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  9. 2017Attention-Based Multimodal Fusion for Video Descriptioncited by 75
  10. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  11. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 71
  12. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 61
  13. 2020Discriminative Multi-modality Speech Recognitioncited by 56
  14. 2022Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Readingcited by 39
  15. 2020Audio-visual Recognition of Overlapped speech for the LRS2 datasetcited by 38
  16. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  17. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  18. 2022Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Datacited by 30
  19. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
  20. 2022Robust Data2vec: Noise-robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learningcited by 21
  21. 2022Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learningcited by 21
  22. 2022A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognitioncited by 20
  23. 2021Interactive Feature Fusion for End-to-End Noise-Robust Speech Recognitioncited by 17
  24. 2021The USTC-NELSLIP Systems for Simultaneous Speech Translation Task at IWSLT 2021cited by 13
  25. 2022Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognitioncited by 12
  26. 2019EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networkscited by 11
  27. 2023Metric-oriented Speech Enhancement using Diffusion Probabilistic Modelcited by 11
  28. 2022Dual-Path Style Learning for End-to-End Noise-Robust Speech Recognitioncited by 10
  29. 2023Unifying Speech Enhancement and Separation with Gradient Modulation for End-to-End Noise-Robust Speech Separationcited by 10
  30. 2023Gradient Remedy for Multi-Task Learning in End-to-End Noise-Robust Speech Recognitioncited by 10
  31. 2023Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASRcited by 10
  32. 2023Unsupervised Noise adaptation using Data Simulationcited by 9
  33. 2022Self-critical Sequence Training for Automatic Speech Recognitioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 1.

9 papers.

  1. 2024A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognitioncited by 12
  2. 2024Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognitioncited by 7
  3. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
  4. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognitioncited by 4
  5. 2025Balance-aware Sequence Sampling Makes Multi-modal Learning Bettercited by 2
  6. 2025Attention-Driven Multimodal Alignment for Long-term Action Quality Assessmentcited by 2
  7. 2025GLip: A Global-Local Integrated Progressive Framework for Robust Visual Speech Recognitioncited by 2
  8. 2025ESG-Net: Event-Aware Semantic Guided Network for Dense Audio-Visual Event Localizationcited by 1
  9. 2026Head-Pose-Aware Visual Speech Recognition with FiLM Modulationcited by 0