mlsanityData to 14 Oct 2026

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

arXiv 2306.10567 ↗First posted 18 Jun 2023eess.AS cs.CV cs.MMcited by 4

Not on the map. Why?

Abstract (arXiv)

Audio-visual speech recognition (AVSR) attracts a surge of research interest recently by leveraging multimodal signals to understand human speech. Mainstream approaches addressing this task have developed sophisticated architectures and techniques for multi-modality fusion and representation learning. However, the natural heterogeneity of different modalities causes distribution gap between their representations, making it challenging to fuse them. In this paper, we aim to learn the shared representations across modalities to bridge their gap. Different from existing similar methods on other multimodal tasks like sentiment analysis, we focus on the temporal contextual dependencies considering the sequence-to-sequence task setting of AVSR. In particular, we propose an adversarial network to refine frame-level modality-invariant representations (MIR-GAN), which captures the commonality across modalities to ease the subsequent multimodal fusion process. Extensive experiments on public benchmarks LRS3 and LRS2 show that our approach outperforms the state-of-the-arts.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 43 of its 72 references that are in the map. First posted on arXiv 18 Jun 2023.

Builds on

43 of 72 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016Layer Normalizationcited by 5,669
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  8. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  10. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  11. 2016Generative Adversarial Text to Image Synthesiscited by 735
  12. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
31 more
  1. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  2. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  3. 2018Deep Audio-Visual Speech Recognitioncited by 322
  4. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
  5. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  6. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 265
  7. 2016Lip Reading Sentences in the Wildcited by 228
  8. 2015Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networkscited by 214
  9. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 180
  10. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 151
  11. 2022Visual Speech Recognition for Multiple Languages in the Wildcited by 95
  12. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 75
  13. 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 74
  14. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 64
  15. 2020Discriminative Multi-modality Speech Recognitioncited by 58
  16. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 39
  17. 2020Audio-visual Recognition of Overlapped speech for the LRS2 datasetcited by 38
  18. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 37
  19. 2022Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Datacited by 30
  20. 2022Robust Data2vec: Noise-robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learningcited by 22
  21. 2022Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learningcited by 22
  22. 2021Interactive Feature Fusion for End-to-End Noise-Robust Speech Recognitioncited by 18
  23. 2022Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognitioncited by 12
  24. 2019EleAtt-RNN: Adding Attentiveness to Neurons in Recurrent Neural Networkscited by 11
  25. 2023Metric-oriented Speech Enhancement using Diffusion Probabilistic Modelcited by 11
  26. 2022Dual-Path Style Learning for End-to-End Noise-Robust Speech Recognitioncited by 10
  27. 2023Unifying Speech Enhancement and Separation with Gradient Modulation for End-to-End Noise-Robust Speech Separationcited by 10
  28. 2023Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASRcited by 10
  29. 2023Unsupervised Noise adaptation using Data Simulationcited by 9
  30. 2021Discriminative Latent Semantic Graph for Video Captioningcited by 7
  31. 2021Exploring Modality-shared Appearance Features and Modality-invariant Relation Features for Cross-modality Person Re-Identificationcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2024MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correctioncited by 8
  2. 2024Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognitioncited by 7
  3. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
  4. 2023Boosting Audio-visual Zero-shot Learning with Large Language Modelscited by 1