# Modality Attention for End-to-End Audio-visual Speech Recognition

arXiv: https://arxiv.org/abs/1811.05250 · HTML: https://mlsanity.com/paper/1811.05250/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2018.

## Built on by (most cited first, 13 of 13)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Deep Spoken Keyword Spotting: An Overview](https://mlsanity.com/paper/2111.10592/) (2021)
- [Data Poisoning Attacks Against Multimodal Encoders](https://mlsanity.com/paper/2209.15266/) (2022)
- [Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning](https://mlsanity.com/paper/2212.05301/) (2022)
- [A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition](https://mlsanity.com/paper/2403.04245/) (2024)
- [Multi-level Attention Fusion Network for Audio-visual Event Recognition](https://mlsanity.com/paper/2106.06736/) (2021)
- [Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition](https://mlsanity.com/paper/2307.02909/) (2023)
- [A Comparative Study of Speaker Role Identification in Air Traffic Communication Using Deep Learning Approaches](https://mlsanity.com/paper/2111.02041/) (2021)
- [Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder](https://mlsanity.com/paper/2308.08488/) (2023)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [On Robustness to Missing Video for Audiovisual Speech Recognition](https://mlsanity.com/paper/2312.10088/) (2023)
- [DM2S2: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention](https://mlsanity.com/paper/2209.03126/) (2022)
- [Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognition](https://mlsanity.com/paper/2608.26213/) (2026)

## Abstract (arXiv)

Audio-visual speech recognition (AVSR) system is thought to be one of the most promising solutions for robust speech recognition, especially in noisy environment. In this paper, we propose a novel multimodal attention based method for audio-visual speech recognition which could automatically learn the fused representation from both modalities based on their importance. Our method is realized using state-of-the-art sequence-to-sequence (Seq2seq) architectures. Experimental results show that relative improvements from 2% up to 36% over the auditory modality alone are obtained depending on the different signal-to-noise-ratio (SNR). Compared to the traditional feature concatenation methods, our proposed approach can achieve better recognition performance under both clean and noisy conditions. We believe modality attention based end-to-end method can be easily generalized to other multimodal tasks with correlated information.