# Cross-Modal Self-Attention with Multi-Task Pre-Training for Medical Visual Question Answering

arXiv: https://arxiv.org/abs/2105.00136 · HTML: https://mlsanity.com/paper/2105.00136/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2021.

## Built on by (most cited first, 26 of 26)

- [Medical Visual Question Answering: A Survey](https://mlsanity.com/paper/2111.10056/) (2021)
- [Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models](https://mlsanity.com/paper/2303.05977/) (2023)
- [Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge](https://mlsanity.com/paper/2209.07118/) (2022)
- [MedCoT: Medical Chain of Thought via Hierarchical Expert](https://mlsanity.com/paper/2412.13736/) (2024)
- [MultiGPrompt for Multi-Task Pre-Training and Prompting on Graphs](https://mlsanity.com/paper/2312.03731/) (2023)
- [RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training](https://mlsanity.com/paper/2303.00534/) (2023)
- [Self-supervised vision-language pretraining for Medical visual question answering](https://mlsanity.com/paper/2211.13594/) (2022)
- [GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosis](https://mlsanity.com/paper/2411.16778/) (2024)
- [Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training](https://mlsanity.com/paper/2305.07920/) (2023)
- [R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest](https://mlsanity.com/paper/2410.20327/) (2024)
- [A Survey of Natural Language Generation](https://mlsanity.com/paper/2112.11739/) (2021)
- [A scoping review on multimodal deep learning in biomedical images and texts](https://mlsanity.com/paper/2307.07362/) (2023)
- [Consistency-preserving Visual Question Answering in Medical Imaging](https://mlsanity.com/paper/2206.13296/) (2022)
- [UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering](https://mlsanity.com/paper/2212.10729/) (2022)
- [Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark](https://mlsanity.com/paper/2306.06494/) (2023)
- [Localized Questions in Medical Visual Question Answering](https://mlsanity.com/paper/2307.01067/) (2023)
- [Free Form Medical Visual Question Answering in Radiology](https://mlsanity.com/paper/2401.13081/) (2024)
- [Prompt-based Personalized Federated Learning for Medical Visual Question Answering](https://mlsanity.com/paper/2402.09677/) (2024)
- [BESTMVQA: A Benchmark Evaluation System for Medical Visual Question Answering](https://mlsanity.com/paper/2312.07867/) (2023)
- [MedReason-R1: Learning to Reason for CT Diagnosis with Reinforcement Learning and Local Zoom](https://mlsanity.com/paper/2510.19626/) (2025)
- [Tri-VQA: Triangular Reasoning Medical Visual Question Answering for Multi-Attribute Analysis](https://mlsanity.com/paper/2406.15050/) (2024)
- [SimpsonsVQA: Enhancing Inquiry-Based Learning with a Tailored Dataset](https://mlsanity.com/paper/2410.22648/) (2024)
- [Computed Tomography Visual Question Answering with Cross-modal Feature Graphing](https://mlsanity.com/paper/2507.04333/) (2025)
- [Barriers in Integrating Medical Visual Question Answering into Radiology Workflows: A Scoping Review and Clinicians' Insights](https://mlsanity.com/paper/2507.08036/) (2025)
- [MF2-MVQA: A Multi-stage Feature Fusion method for Medical Visual Question Answering](https://mlsanity.com/paper/2211.05991/) (2022)
- [Semantically Calibrated Evidence Composition for CT Vision-Language Learning](https://mlsanity.com/paper/2608.00239/) (2026)

## Abstract (arXiv)

Due to the severe lack of labeled data, existing methods of medical visual question answering usually rely on transfer learning to obtain effective image feature representation and use cross-modal fusion of visual and linguistic features to achieve question-related answer prediction. These two phases are performed independently and without considering the compatibility and applicability of the pre-trained features for cross-modal fusion. Thus, we reformulate image feature pre-training as a multi-task learning paradigm and witness its extraordinary superiority, forcing it to take into account the applicability of features for the specific image comprehension task. Furthermore, we introduce a cross-modal self-attention~(CMSA) module to selectively capture the long-range contextual relevance for more effective fusion of visual and linguistic features. Experimental results demonstrate that the proposed method outperforms existing state-of-the-art methods. Our code and models are available at https://github.com/haifangong/CMSA-MTPT-4-MedicalVQA.