# RJUA-MedDQA: A Multimodal Benchmark for Medical Document Question Answering and Clinical Reasoning

arXiv: https://arxiv.org/abs/2402.14840 · HTML: https://mlsanity.com/paper/2402.14840/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Clinical and biomedical applications, one of 17 directions in LLM reasoning.
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Placed in

- LLM reasoning / [Clinical and biomedical applications](https://mlsanity.com/lr/clinical-and-biomedical-applications/)

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [PathVQA: 30000+ Questions for Medical Visual Question Answering](https://mlsanity.com/paper/2003.10286/) (2020)
- [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://mlsanity.com/paper/2102.09542/) (2021)
- [InfographicVQA](https://mlsanity.com/paper/2104.12756/) (2021)
- [MultiModalQA: Complex Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2104.06039/) (2021)
- [SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images](https://mlsanity.com/paper/2301.04883/) (2023)
- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [VisualMRC: Machine Reading Comprehension on Document Images](https://mlsanity.com/paper/2101.11272/) (2021)
- [ICDAR 2019 Competition on Scene Text Visual Question Answering](https://mlsanity.com/paper/1907.00490/) (2019)
- [PDFVQA: A New Dataset for Real-World VQA on PDF Documents](https://mlsanity.com/paper/2304.06447/) (2023)
- ["My nose is running.""Are you also coughing?": Building A Medical Diagnosis Agent with Interpretable Inquiry Logics](https://mlsanity.com/paper/2204.13953/) (2022)
- [Context-VQA: Towards Context-Aware and Purposeful Visual Question Answering](https://mlsanity.com/paper/2307.15745/) (2023)
- [Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark](https://mlsanity.com/paper/2306.06494/) (2023)

## Built on by (most cited first, 8 of 8)

- [PoTable: Towards Systematic Thinking via Plan-then-Execute Stage Reasoning on Tables](https://mlsanity.com/paper/2412.04272/) (2024)
- [Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applications](https://mlsanity.com/paper/2508.00669/) (2025)
- [From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice](https://mlsanity.com/paper/2410.01812/) (2024)
- [MedPAIR: Measuring Physicians and AI Relevance Alignment in Medical Question Answering](https://mlsanity.com/paper/2505.24040/) (2025)
- [VideoConviction: A Multimodal Benchmark for Human Conviction and Stock Market Recommendations](https://mlsanity.com/paper/2507.08104/) (2025)
- [ZALM3: Zero-Shot Enhancement of Vision-Language Alignment via In-Context Information in Multi-Turn Multimodal Medical Dialogue](https://mlsanity.com/paper/2409.17610/) (2024)
- [Medusa: Cross-Modal Transferable Adversarial Attacks on Multimodal Medical Retrieval-Augmented Generation](https://mlsanity.com/paper/2511.19257/) (2025)
- [MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis](https://mlsanity.com/paper/2608.22323/) (2026)

## Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) and Large Multi-modal Models (LMMs) have shown potential in various medical applications, such as Intelligent Medical Diagnosis. Although impressive results have been achieved, we find that existing benchmarks do not reflect the complexity of real medical reports and specialized in-depth reasoning capabilities. In this work, we introduced RJUA-MedDQA, a comprehensive benchmark in the field of medical specialization, which poses several challenges: comprehensively interpreting imgage content across diverse challenging layouts, possessing numerical reasoning ability to identify abnormal indicators and demonstrating clinical reasoning ability to provide statements of disease diagnosis, status and advice based on medical contexts. We carefully design the data generation pipeline and proposed the Efficient Structural Restoration Annotation (ESRA) Method, aimed at restoring textual and tabular content in medical report images. This method substantially enhances annotation efficiency, doubling the productivity of each annotator, and yields a 26.8% improvement in accuracy. We conduct extensive evaluations, including few-shot assessments of 5 LMMs which are capable of solving Chinese medical QA tasks. To further investigate the limitations and potential of current LMMs, we conduct comparative experiments on a set of strong LLMs by using image-text generated by ESRA method. We report the performance of baselines and offer several observations: (1) The overall performance of existing LMMs is still limited; however LMMs more robust to low-quality and diverse-structured images compared to LLMs. (3) Reasoning across context and image content present significant challenges. We hope this benchmark helps the community make progress on these challenging tasks in multi-modal medical document understanding and facilitate its application in healthcare.