# Multimodal Unified Attention Networks for Vision-and-Language Interactions

arXiv: https://arxiv.org/abs/1908.04107 · HTML: https://mlsanity.com/paper/1908.04107/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Aug 2019.

## Built on by (most cited first, 9 of 9)

- [Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks](https://mlsanity.com/paper/2204.07780/) (2022)
- [A Multimodal Approach for Dementia Detection from Spontaneous Speech with Tensor Fusion Layer](https://mlsanity.com/paper/2211.04368/) (2022)
- [Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech](https://mlsanity.com/paper/2305.16406/) (2023)
- [EAML: Ensemble Self-Attention-based Mutual Learning Network for Document Image Classification](https://mlsanity.com/paper/2305.06923/) (2023)
- [Latent Alignment of Procedural Concepts in Multimodal Recipes](https://mlsanity.com/paper/2101.04727/) (2021)
- [Transformer-based Personalized Attention Mechanism for Medical Images with Clinical Records](https://mlsanity.com/paper/2206.03003/) (2022)
- [Bilaterally Slimmable Transformer for Elastic and Efficient Visual Question Answering](https://mlsanity.com/paper/2203.12814/) (2022)
- [ViASNet: A Video Ad Saliency Network for Predicting Dynamic Saliency and Viewer Engagement](https://mlsanity.com/paper/2605.29302/) (2026)
- [Alzheimer's Disease Diagnosis using a Multimodal Approach with 3D MRI and PET](https://mlsanity.com/paper/2606.20037/) (2026)

## Abstract (arXiv)

Learning an effective attention mechanism for multimodal data is important in many vision-and-language tasks that require a synergic understanding of both the visual and textual contents. Existing state-of-the-art approaches use co-attention models to associate each visual object (e.g., image region) with each textual object (e.g., query word). Despite the success of these co-attention models, they only model inter-modal interactions while neglecting intra-modal interactions. Here we propose a general `unified attention' model that simultaneously captures the intra- and inter-modal interactions of multimodal features and outputs their corresponding attended representations. By stacking such unified attention blocks in depth, we obtain the deep Multimodal Unified Attention Network (MUAN), which can seamlessly be applied to the visual question answering (VQA) and visual grounding tasks. We evaluate our MUAN models on two VQA datasets and three visual grounding datasets, and the results show that MUAN achieves top-level performance on both tasks without bells and whistles.