# Multi-task Learning Of Deep Neural Networks For Audio Visual Automatic Speech Recognition

arXiv: https://arxiv.org/abs/1701.02477 · HTML: https://mlsanity.com/paper/1701.02477/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jan 2017.

## Built on by (most cited first, 2 of 2)

- [Query-Response Interactions by Multi-tasks in Semantic Search for Chatbot Candidate Retrieval](https://mlsanity.com/paper/2208.11018/) (2022)
- [A Multi-task Learning Balanced Attention Convolutional Neural Network Model for Few-shot Underwater Acoustic Target Recognition](https://mlsanity.com/paper/2504.13102/) (2025)

## Abstract (arXiv)

Multi-task learning (MTL) involves the simultaneous training of two or more related tasks over shared representations. In this work, we apply MTL to audio-visual automatic speech recognition(AV-ASR). Our primary task is to learn a mapping between audio-visual fused features and frame labels obtained from acoustic GMM/HMM model. This is combined with an auxiliary task which maps visual features to frame labels obtained from a separate visual GMM/HMM model. The MTL model is tested at various levels of babble noise and the results are compared with a base-line hybrid DNN-HMM AV-ASR model. Our results indicate that MTL is especially useful at higher level of noise. Compared to base-line, upto 7% relative improvement in WER is reported at -3 SNR dB