# Investigations on End-to-End Audiovisual Fusion

arXiv: https://arxiv.org/abs/1804.11127 · HTML: https://mlsanity.com/paper/1804.11127/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2018.

## Built on by (most cited first, 3 of 3)

- [LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformers](https://mlsanity.com/paper/2302.02141/) (2023)
- [A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation](https://mlsanity.com/paper/2308.08849/) (2023)
- [Investigations on Audiovisual Emotion Recognition in Noisy Conditions](https://mlsanity.com/paper/2103.01894/) (2021)

## Abstract (arXiv)

Audiovisual speech recognition (AVSR) is a method to alleviate the adverse effect of noise in the acoustic signal. Leveraging recent developments in deep neural network-based speech recognition, we present an AVSR neural network architecture which is trained end-to-end, without the need to separately model the process of decision fusion as in conventional (e.g. HMM-based) systems. The fusion system outperforms single-modality recognition under all noise conditions. Investigation of the saliency of the input features shows that the neural network automatically adapts to different noise levels in the acoustic signal.