# LaPA: Latent Prompt Assist Model For Medical Visual Question Answering

arXiv: https://arxiv.org/abs/2404.13039 · HTML: https://mlsanity.com/paper/2404.13039/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 19 Apr 2024.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Stacked Attention Networks for Image Question Answering](https://mlsanity.com/paper/1511.02274/) (2015)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://mlsanity.com/paper/2109.04332/) (2021)
- [Interactive and Explainable Region-guided Radiology Report Generation](https://mlsanity.com/paper/2304.08295/) (2023)
- [Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training](https://mlsanity.com/paper/2209.07098/) (2022)
- [Medical Visual Question Answering: A Survey](https://mlsanity.com/paper/2111.10056/) (2021)
- [Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering](https://mlsanity.com/paper/2303.05352/) (2023)
- [Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?](https://mlsanity.com/paper/2112.13906/) (2021)
- [Overcoming Data Limitation in Medical Visual Question Answering](https://mlsanity.com/paper/1909.11867/) (2019)
- [MedICaT: A Dataset of Medical Images, Captions, and Textual References](https://mlsanity.com/paper/2010.06000/) (2020)
- [MMBERT: Multimodal BERT Pretraining for Improved Medical VQA](https://mlsanity.com/paper/2104.01394/) (2021)
- [Multiple Meta-model Quantifying for Medical Visual Question Answering](https://mlsanity.com/paper/2105.08913/) (2021)
- [Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge](https://mlsanity.com/paper/2209.07118/) (2022)
- [Q2ATransformer: Improving Medical VQA via an Answer Querying Decoder](https://mlsanity.com/paper/2304.01611/) (2023)
- [Self-supervised vision-language pretraining for Medical visual question answering](https://mlsanity.com/paper/2211.13594/) (2022)
- [Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answering](https://mlsanity.com/paper/2307.05314/) (2023)
- [Complex Organ Mask Guided Radiology Report Generation](https://mlsanity.com/paper/2311.02329/) (2023)
- [Latent Prompt Tuning for Text Summarization](https://mlsanity.com/paper/2211.01837/) (2022)

## Built on by (most cited first, 12 of 12)

- [Multimodal Large Language Models for Medicine: A Comprehensive Survey](https://mlsanity.com/paper/2504.21051/) (2025)
- [MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution](https://mlsanity.com/paper/2604.26283/) (2026)
- [A Survey of Medical Vision-and-Language Applications and Their Techniques](https://mlsanity.com/paper/2411.12195/) (2024)
- [Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopy](https://mlsanity.com/paper/2506.09958/) (2025)
- [ORID: Organ-Regional Information Driven Framework for Radiology Report Generation](https://mlsanity.com/paper/2411.13025/) (2024)
- [A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answering](https://mlsanity.com/paper/2508.02841/) (2025)
- [Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering](https://mlsanity.com/paper/2510.08791/) (2025)
- [Q-FSRU: Quantum-Augmented Frequency-Spectral Fusion for Medical Visual Question Answering](https://mlsanity.com/paper/2508.12036/) (2025)
- [Q-FSRU: Quantum-Augmented Frequency-Spectral For Medical Visual Question Answering](https://mlsanity.com/paper/2509.23899/) (2025)
- [CMI-MTL: Cross-Mamba interaction based multi-task learning for medical visual question answering](https://mlsanity.com/paper/2511.01357/) (2025)
- [InViC: Intent-aware Visual Cues for Medical Visual Question Answering](https://mlsanity.com/paper/2603.16372/) (2026)
- [KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answering](https://mlsanity.com/paper/2604.00601/) (2026)

## Abstract (arXiv)

Medical visual question answering (Med-VQA) aims to automate the prediction of correct answers for medical images and questions, thereby assisting physicians in reducing repetitive tasks and alleviating their workload. Existing approaches primarily focus on pre-training models using additional and comprehensive datasets, followed by fine-tuning to enhance performance in downstream tasks. However, there is also significant value in exploring existing models to extract clinically relevant information. In this paper, we propose the Latent Prompt Assist model (LaPA) for medical visual question answering. Firstly, we design a latent prompt generation module to generate the latent prompt with the constraint of the target answer. Subsequently, we propose a multi-modal fusion block with latent prompt fusion module that utilizes the latent prompt to extract clinical-relevant information from uni-modal and multi-modal features. Additionally, we introduce a prior knowledge fusion module to integrate the relationship between diseases and organs with the clinical-relevant information. Finally, we combine the final integrated information with image-language cross-modal information to predict the final answers. Experimental results on three publicly available Med-VQA datasets demonstrate that LaPA outperforms the state-of-the-art model ARL, achieving improvements of 1.83%, 0.63%, and 1.80% on VQA-RAD, SLAKE, and VQA-2019, respectively. The code is publicly available at https://github.com/GaryGuTC/LaPA_model.