# Deep Multimodal Learning for Audio-Visual Speech Recognition

arXiv: https://arxiv.org/abs/1501.05396 · HTML: https://mlsanity.com/paper/1501.05396/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jan 2015.

## Built on by (most cited first, 30 of 38)

- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Provable Dynamic Fusion for Low-Quality Multimodal Data](https://mlsanity.com/paper/2306.02050/) (2023)
- [A Survey on Multimodal Disinformation Detection](https://mlsanity.com/paper/2103.12541/) (2021)
- [Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring](https://mlsanity.com/paper/2303.08536/) (2023)
- [Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization](https://mlsanity.com/paper/2109.02401/) (2021)
- [Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition](https://mlsanity.com/paper/2207.06020/) (2022)
- [Deep Learning for Visual Speech Analysis: A Survey](https://mlsanity.com/paper/2205.10839/) (2022)
- [MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens](https://mlsanity.com/paper/2503.11315/) (2025)
- [Emotion Based Hate Speech Detection using Multimodal Learning](https://mlsanity.com/paper/2202.06218/) (2022)
- [Audio-Visual Transformer Based Crowd Counting](https://mlsanity.com/paper/2109.01926/) (2021)
- [Deep Learning for Technical Document Classification](https://mlsanity.com/paper/2106.14269/) (2021)
- [XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception](https://mlsanity.com/paper/2403.14402/) (2024)
- [Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillation](https://mlsanity.com/paper/2212.02782/) (2022)
- [Lagrangian Motion Magnification with Double Sparse Optical Flow Decomposition](https://mlsanity.com/paper/2204.07636/) (2022)
- [Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition](https://mlsanity.com/paper/2407.03563/) (2024)
- [Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables](https://mlsanity.com/paper/2505.12473/) (2025)
- [Alternative Telescopic Displacement: An Efficient Multimodal Alignment Method](https://mlsanity.com/paper/2306.16950/) (2023)
- [Is Lip Region-of-Interest Sufficient for Lipreading?](https://mlsanity.com/paper/2205.14295/) (2022)
- [Zoom and Shift are All You Need](https://mlsanity.com/paper/2406.08866/) (2024)
- [Advances and Challenges in Deep Lip Reading](https://mlsanity.com/paper/2110.07879/) (2021)
- [PoseKernelLifter: Metric Lifting of 3D Human Pose using Sound](https://mlsanity.com/paper/2112.00216/) (2021)
- [Deep Equilibrium Multimodal Fusion](https://mlsanity.com/paper/2306.16645/) (2023)
- [Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition](https://mlsanity.com/paper/2307.02909/) (2023)
- [Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation](https://mlsanity.com/paper/2501.07110/) (2025)
- [MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2510.04136/) (2025)
- [Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models](https://mlsanity.com/paper/2511.07253/) (2025)
- [Audio-visual multi-channel speech separation, dereverberation and recognition](https://mlsanity.com/paper/2204.01977/) (2022)
- [Multimodal Learning for Multi-Omics: A Survey](https://mlsanity.com/paper/2211.16509/) (2022)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)

## Abstract (arXiv)

In this paper, we present methods in deep multimodal learning for fusing speech and visual modalities for Audio-Visual Automatic Speech Recognition (AV-ASR). First, we study an approach where uni-modal deep networks are trained separately and their final hidden layers fused to obtain a joint feature space in which another deep network is built. While the audio network alone achieves a phone error rate (PER) of 41% under clean condition on the IBM large vocabulary audio-visual studio dataset, this fusion model achieves a PER of 35.83% demonstrating the tremendous value of the visual channel in phone classification even in audio with high signal to noise ratio. Second, we present a new deep network architecture that uses a bilinear softmax layer to account for class specific correlations between modalities. We show that combining the posteriors from the bilinear networks with those from the fused model mentioned above results in a further significant phone error rate reduction, yielding a final PER of 34.03%.