# Multi-modality Latent Interaction Network for Visual Question Answering

arXiv: https://arxiv.org/abs/1908.04289 · HTML: https://mlsanity.com/paper/1908.04289/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Aug 2019.

## Built on by (most cited first, 23 of 23)

- [Fast Convergence of DETR with Spatially Modulated Co-Attention](https://mlsanity.com/paper/2101.07448/) (2021)
- [Visuo-Tactile Transformers for Manipulation](https://mlsanity.com/paper/2210.00121/) (2022)
- [Container: Context Aggregation Network](https://mlsanity.com/paper/2106.01401/) (2021)
- [Greedy Gradient Ensemble for Robust Visual Question Answering](https://mlsanity.com/paper/2107.12651/) (2021)
- [Dual-stream Network for Visual Recognition](https://mlsanity.com/paper/2105.14734/) (2021)
- [Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks](https://mlsanity.com/paper/2204.07780/) (2022)
- [Oriented Object Detection with Transformer](https://mlsanity.com/paper/2106.03146/) (2021)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [General Greedy De-bias Learning](https://mlsanity.com/paper/2112.10572/) (2021)
- [Interactive Question Answering Systems: Literature Review](https://mlsanity.com/paper/2209.01621/) (2022)
- [Latent Variable Models for Visual Question Answering](https://mlsanity.com/paper/2101.06399/) (2021)
- [Scalable Transformers for Neural Machine Translation](https://mlsanity.com/paper/2106.02242/) (2021)
- [SceneGATE: Scene-Graph based co-Attention networks for TExt visual question answering](https://mlsanity.com/paper/2212.08283/) (2022)
- [Object Attribute Matters in Visual Question Answering](https://mlsanity.com/paper/2401.09442/) (2023)
- [Listen Then See: Video Alignment with Speaker Attention](https://mlsanity.com/paper/2404.13530/) (2024)
- [Question-Driven Graph Fusion Network For Visual Question Answering](https://mlsanity.com/paper/2204.00975/) (2022)
- [Visual Question Generation in Bengali](https://mlsanity.com/paper/2310.08187/) (2023)
- [What is Multimodality?](https://mlsanity.com/paper/2103.06304/) (2021)
- [Fast Convergence of DETR with Spatially Modulated Co-Attention](https://mlsanity.com/paper/2108.02404/) (2021)
- [On the Significance of Question Encoder Sequence Model in the Out-of-Distribution Performance in Visual Question Answering](https://mlsanity.com/paper/2108.12585/) (2021)
- [Few-Shot Visual Question Generation: A Novel Task and Benchmark Datasets](https://mlsanity.com/paper/2210.07076/) (2022)
- [VQA with Cascade of Self- and Co-Attention Blocks](https://mlsanity.com/paper/2302.14777/) (2023)
- [A Pattern to Align Them All: Integrating Different Modalities to Define Multi-Modal Entities](https://mlsanity.com/paper/2410.13803/) (2024)

## Abstract (arXiv)

Exploiting relationships between visual regions and question words have achieved great success in learning multi-modality features for Visual Question Answering (VQA). However, we argue that existing methods mostly model relations between individual visual regions and words, which are not enough to correctly answer the question. From humans' perspective, answering a visual question requires understanding the summarizations of visual and language information. In this paper, we proposed the Multi-modality Latent Interaction module (MLI) to tackle this problem. The proposed module learns the cross-modality relationships between latent visual and language summarizations, which summarize visual regions and question into a small number of latent representations to avoid modeling uninformative individual region-word relations. The cross-modality information between the latent summarizations are propagated to fuse valuable information from both modalities and are used to update the visual and word features. Such MLI modules can be stacked for several stages to model complex and latent relations between the two modalities and achieves highly competitive performance on public VQA benchmarks, VQA v2.0 and TDIUC . In addition, we show that the performance of our methods could be significantly improved by combining with pre-trained language model BERT.