# Multiresolution and Multimodal Speech Recognition with Transformers

arXiv: https://arxiv.org/abs/2004.14840 · HTML: https://mlsanity.com/paper/2004.14840/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 10 of 10)

- [LocalTrans: A Multiscale Local Transformer Network for Cross-Resolution Homography Estimation](https://mlsanity.com/paper/2106.04067/) (2021)
- [AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR](https://mlsanity.com/paper/2303.16501/) (2023)
- [AVATAR: Unconstrained Audiovisual Speech Recognition](https://mlsanity.com/paper/2206.07684/) (2022)
- [Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greek](https://mlsanity.com/paper/2301.00304/) (2022)
- [MMLatch: Bottom-up Top-down Fusion for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2201.09828/) (2022)
- [Robust Audiovisual Speech Recognition Models with Mixture-of-Experts](https://mlsanity.com/paper/2409.12370/) (2024)
- [ASR-Aware End-to-end Neural Diarization](https://mlsanity.com/paper/2202.01286/) (2022)
- [SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data](https://mlsanity.com/paper/2408.00624/) (2024)
- [Adaptation and Optimization of Automatic Speech Recognition (ASR) for the Maritime Domain in the Field of VHF Communication](https://mlsanity.com/paper/2306.00614/) (2023)
- [Transformers for prompt-level EMA non-response prediction](https://mlsanity.com/paper/2111.01193/) (2021)

## Abstract (arXiv)

This paper presents an audio visual automatic speech recognition (AV-ASR) system using a Transformer-based architecture. We particularly focus on the scene context provided by the visual information, to ground the ASR. We extract representations for audio features in the encoder layers of the transformer and fuse video features using an additional crossmodal multihead attention layer. Additionally, we incorporate a multitask training criterion for multiresolution ASR, where we train the model to generate both character and subword level transcriptions. Experimental results on the How2 dataset, indicate that multiresolution training can speed up convergence by around 50% and relatively improves word error rate (WER) performance by upto 18% over subword prediction models. Further, incorporating visual information improves performance with relative gains upto 3.76% over audio only models. Our results are comparable to state-of-the-art Listen, Attend and Spell-based architectures.