# MuVAM: A Multi-View Attention-based Model for Medical Visual Question Answering

arXiv: https://arxiv.org/abs/2107.03216 · HTML: https://mlsanity.com/paper/2107.03216/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 43 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 29 of 29)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [MUTAN: Multimodal Tucker Fusion for Visual Question Answering](https://mlsanity.com/paper/1705.06676/) (2017)
- [Relation-Aware Graph Attention Network for Visual Question Answering](https://mlsanity.com/paper/1903.12314/) (2019)
- [Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering](https://mlsanity.com/paper/1708.01471/) (2017)
- [Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering](https://mlsanity.com/paper/1708.03619/) (2017)
- [Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge](https://mlsanity.com/paper/1708.02711/) (2017)
- [Overcoming Data Limitation in Medical Visual Question Answering](https://mlsanity.com/paper/1909.11867/) (2019)
- [Multiple Meta-model Quantifying for Medical Visual Question Answering](https://mlsanity.com/paper/2105.08913/) (2021)
- [On the Effect of Inter-observer Variability for a Reliable Estimation of Uncertainty of Medical Image Segmentation](https://mlsanity.com/paper/1806.02562/) (2018)
- [Image Captioning and Visual Question Answering Based on Attributes and External Knowledge](https://mlsanity.com/paper/1603.02814/) (2016)
- [Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources](https://mlsanity.com/paper/1511.06973/) (2015)
- [Simple Baseline for Visual Question Answering](https://mlsanity.com/paper/1512.02167/) (2015)
- [A Question-Centric Model for Visual Question Answering in Medical Imaging](https://mlsanity.com/paper/2003.08760/) (2020)
- [Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions](https://mlsanity.com/paper/1801.09041/) (2018)
- [Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks](https://mlsanity.com/paper/1712.00733/) (2017)
- [The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions](https://mlsanity.com/paper/1612.05386/) (2016)
- [Partial Policy-based Reinforcement Learning for Anatomical Landmark Localization in 3D Medical Images](https://mlsanity.com/paper/1807.02908/) (2018)
- [DualNet: Domain-Invariant Network for Visual Question Answering](https://mlsanity.com/paper/1606.06108/) (2016)
- [Regularizing Attention Networks for Anomaly Detection in Visual Question Answering](https://mlsanity.com/paper/2009.10054/) (2020)
- [Deep Learning Approaches for Extracting Adverse Events and Indications of Dietary Supplements from Clinical Text](https://mlsanity.com/paper/2009.07780/) (2020)
- [Computer-Aided Diagnosis of Label-Free 3-D Optical Coherence Microscopy Images of Human Cervical Tissue](https://mlsanity.com/paper/1809.10196/) (2018)

## Built on by (most cited first, 4 of 4)

- [Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?](https://mlsanity.com/paper/2112.13906/) (2021)
- [A scoping review on multimodal deep learning in biomedical images and texts](https://mlsanity.com/paper/2307.07362/) (2023)
- [A Dual-Attention Learning Network with Word and Sentence Embedding for Medical Visual Question Answering](https://mlsanity.com/paper/2210.00220/) (2022)
- [SparrowVQE: Visual Question Explanation for Course Content Understanding](https://mlsanity.com/paper/2411.07516/) (2024)

## Abstract (arXiv)

Medical Visual Question Answering (VQA) is a multi-modal challenging task widely considered by research communities of the computer vision and natural language processing. Since most current medical VQA models focus on visual content, ignoring the importance of text, this paper proposes a multi-view attention-based model(MuVAM) for medical visual question answering which integrates the high-level semantics of medical images on the basis of text description. Firstly, different methods are utilized to extract the features of the image and the question for the two modalities of vision and text. Secondly, this paper proposes a multi-view attention mechanism that include Image-to-Question (I2Q) attention and Word-to-Text (W2T) attention. Multi-view attention can correlate the question with image and word in order to better analyze the question and get an accurate answer. Thirdly, a composite loss is presented to predict the answer accurately after multi-modal feature fusion and improve the similarity between visual and textual cross-modal features. It consists of classification loss and image-question complementary (IQC) loss. Finally, for data errors and missing labels in the VQA-RAD dataset, we collaborate with medical experts to correct and complete this dataset and then construct an enhanced dataset, VQA-RADPh. The experiments on these two datasets show that the effectiveness of MuVAM surpasses the state-of-the-art method.