mlsanityData to 14 Oct 2026

Deep Multimodal Learning for Audio-Visual Speech Recognition

arXiv 1501.05396 ↗First posted 22 Jan 2015cs.CL cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

In this paper, we present methods in deep multimodal learning for fusing speech and visual modalities for Audio-Visual Automatic Speech Recognition (AV-ASR). First, we study an approach where uni-modal deep networks are trained separately and their final hidden layers fused to obtain a joint feature space in which another deep network is built. While the audio network alone achieves a phone error rate (PER) of 41% under clean condition on the IBM large vocabulary audio-visual studio dataset, this fusion model achieves a PER of 35.83% demonstrating the tremendous value of the visual channel in phone classification even in audio with high signal to noise ratio. Second, we present a new deep network architecture that uses a bilinear softmax layer to account for class specific correlations between modalities. We show that combining the posteriors from the bilinear networks with those from the fused model mentioned above results in a further significant phone error rate reduction, yielding a final PER of 34.03%.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Jan 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 6; 2022: 10; 2023: 7; 2024: 4; 2025: 10; 2026: 1.

38 papers.

Sort:

By area:

  1. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  2. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 116
  3. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 82
  4. 2021A Survey on Multimodal Disinformation Detectioncited by 47
  5. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
  6. 2021Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarizationcited by 32
  7. 2022Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognitioncited by 24
  8. 2022Deep Learning for Visual Speech Analysis: A Surveycited by 21
  9. 2025MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokenscited by 19
  10. 2022Emotion Based Hate Speech Detection using Multimodal Learningcited by 18
  11. 2021Audio-Visual Transformer Based Crowd Countingcited by 12
  12. 2021Deep Learning for Technical Document Classificationcited by 11
26 more
  1. 2024XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perceptioncited by 11
  2. 2022Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillationcited by 9
  3. 2022Lagrangian Motion Magnification with Double Sparse Optical Flow Decompositioncited by 7
  4. 2024Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognitioncited by 7
  5. 2025Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variablescited by 7
  6. 2023Alternative Telescopic Displacement: An Efficient Multimodal Alignment Methodcited by 6
  7. 2022Is Lip Region-of-Interest Sufficient for Lipreading?cited by 5
  8. 2024Zoom and Shift are All You Needcited by 5
  9. 2021Advances and Challenges in Deep Lip Readingcited by 4
  10. 2021PoseKernelLifter: Metric Lifting of 3D Human Pose using Soundcited by 4
  11. 2023Deep Equilibrium Multimodal Fusioncited by 4
  12. 2023Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognitioncited by 4
  13. 2025Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendationcited by 4
  14. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  15. 2025Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Modelscited by 4
  16. 2022Audio-visual multi-channel speech separation, dereverberation and recognitioncited by 3
  17. 2022Multimodal Learning for Multi-Omics: A Surveycited by 3
  18. 2023A Survey on Deep Multi-modal Learning for Body Language Recognition and Generationcited by 3
  19. 2025FedMobile: Enabling Knowledge Contribution-aware Multi-modal Federated Learning with Incomplete Modalitiescited by 3
  20. 2025TimeSense:Making Large Language Models Proficient in Time-Series Analysiscited by 3
  21. 2025Expressive and Scalable Quantum Fusion for Multimodal Learningcited by 2
  22. 2025Efficient Audiovisual Speech Processing via MUTUD: Multimodal Training and Unimodal Deploymentcited by 1
  23. 2025A multi-head deep fusion model for recognition of cattle foraging events using sound and movement signalscited by 1
  24. 2026Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognitioncited by 1
  25. 2022Challenges and Opportunities in Multi-device Speech Processingcited by 0
  26. 2024FM-AE: Frequency-masked Multimodal Autoencoder for Zinc Electrolysis Plate Contact Abnormality Detectioncited by 0