mlsanityData to 14 Oct 2026

Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering

arXiv 2307.11986 ↗First posted 22 Jul 2023cs.CV cs.LGcited by 41

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

To contribute to automating the medical vision-language model, we propose a novel Chest-Xray Difference Visual Question Answering (VQA) task. Given a pair of main and reference images, this task attempts to answer several questions on both diseases and, more importantly, the differences between them. This is consistent with the radiologist's diagnosis practice that compares the current image with the reference before concluding the report. We collect a new dataset, namely MIMIC-Diff-VQA, including 700,703 QA pairs from 164,324 pairs of main and reference images. Compared to existing medical VQA datasets, our questions are tailored to the Assessment-Diagnosis-Intervention-Evaluation treatment procedure used by clinical professionals. Meanwhile, we also propose a novel expert knowledge-aware graph representation learning model to address this task. The proposed baseline model leverages expert knowledge such as anatomical structure prior, semantic, and spatial knowledge to construct a multi-relationship graph, representing the image differences between two images for the image difference VQA task. The dataset and code can be found at https://github.com/Holipori/MIMIC-Diff-VQA. We believe this work would further push forward the medical vision language model.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 37 references that are in the map. First posted on arXiv 22 Jul 2023.

Builds on

21 of 37 references.

References · 21

  1. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  4. 2020Supervised Contrastive Learningcited by 2,829
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  6. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  7. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  8. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 972
  9. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  10. 2019ScispaCy: Fast and Robust Models for Biomedical Natural Language Processingcited by 244
  11. 2020VinDr-CXR: An open dataset of chest X-rays with radiologist's annotationscited by 239
  12. 2018Exploring Visual Relationship for Image Captioningcited by 215
9 more
  1. 2020When Radiology Report Generation Meets Knowledge Graphcited by 145
  2. 2019Relation-Aware Graph Attention Network for Visual Question Answeringcited by 119
  3. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  4. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 90
  5. 2021Describing and Localizing Multiple Changes with Transformerscited by 47
  6. 2018Learning Conditioned Graph Structures for Interpretable Visual Question Answeringcited by 45
  7. 2022Image Difference Captioning with Pre-training and Contrastive Learningcited by 39
  8. 2019Neural Naturalist: Generating Fine-Grained Image Comparisonscited by 36
  9. 2021Automated Generation of Accurate & Fluent Medical X-ray Reportscited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 14; 2026: 13.

41 papers.

Sort:

By area:

  1. 2024RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysiscited by 42
  2. 2024GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosiscited by 29
  3. 2024Libra: Leveraging Temporal Images for Biomedical Radiology Analysiscited by 29
  4. 2024Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Surveycited by 20
  5. 2025ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understandingcited by 20
  6. 2024Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlookscited by 18
  7. 2024M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretationcited by 17
  8. 2023Learning A Multi-Task Transformer Via Unified And Customized Instruction Tuning For Chest Radiograph Interpretationcited by 14
  9. 2025Video Action Differencingcited by 14
  10. 2024Uncertainty-aware Medical Diagnostic Phrase Identification and Groundingcited by 13
  11. 2024Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rayscited by 9
  12. 2024Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: a data-driven approach for improved classificationcited by 8
29 more
  1. 2024D-Rax: Domain-specific Radiologic assistant leveraging multi-modal data and eXpert model predictionscited by 7
  2. 2025A Chain of Diagnosis Framework for Accurate and Explainable Radiology Report Generationcited by 7
  3. 2024Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Modelcited by 5
  4. 2025UniVRSE: Unified Vision-conditioned Response Semantic Entropy for Hallucination Detection in Medical Vision-Language Modelscited by 5
  5. 2025CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationalecited by 5
  6. 2026A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretationcited by 5
  7. 2025Grounding Chest X-Ray Visual Question Answering with Generated Radiology Reportscited by 3
  8. 2024TAB: Transformer Attention Bottlenecks enable User Intervention and Debugging in Vision-Language Modelscited by 2
  9. 2024Expert-level vision-language foundation model for real-world radiology and comprehensive evaluationcited by 1
  10. 2025ChestGPT: Integrating Large Language Models and Vision Transformers for Disease Detection and Localization in Chest X-Rayscited by 1
  11. 2025Computed Tomography Visual Question Answering with Cross-modal Feature Graphingcited by 1
  12. 2025Barriers in Integrating Medical Visual Question Answering into Radiology Workflows: A Scoping Review and Clinicians' Insightscited by 1
  13. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 1
  14. 2026LUMEN: Longitudinal Multi-Modal Radiology Model for Prognosis and Diagnosiscited by 1
  15. 2026An Interpretable Local Editing Model for Counterfactual Medical Image Generationcited by 1
  16. 2026Location-Aware Pretraining for Medical Difference Visual Question Answeringcited by 1
  17. 2026VGS-Decoding: Visual Grounding Score Guided Decoding for Hallucination Mitigation in Medical VLMscited by 1
  18. 2026Enhancing Medical Visual Grounding via Knowledge-guided Spatial Promptscited by 1
  19. 2026DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?cited by 1
  20. 2026A Vision-language Framework for Comparative Reasoning in Radiologycited by 1
  21. 2026CheXpercept: A Benchmark for Evaluating Expert-Level Lesion Perception in Chest X-rayscited by 1
  22. 2025Interpreting Chest X-rays Like a Radiologist: A Benchmark with Clinical Reasoningcited by 0
  23. 2025Saliency Guided Longitudinal Medical Visual Question Answeringcited by 0
  24. 2025MedGEN-Bench: A Contextually Entangled Benchmark for Open-ended Multimodal Medical Generationcited by 0
  25. 2025CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistencycited by 0
  26. 2026Standardizing Longitudinal Radiology Report Evaluation via Large Language Model Annotationcited by 0
  27. 2026MEDIC-AD: Towards Medical Vision-Language Model's Clinical Intelligencecited by 0
  28. 2026Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Modelscited by 0
  29. 2026REVA-PO: Stabilizing Reinforcement Learning for Chest X-ray Report Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Interpretable Medical Image Visual Question Answering via Multi-Modal Relationship Graph Learning
  2. 2026KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answering
  3. 2025ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Images
  4. 2025RadImageNet-VQA: A Large-Scale CT and MRI Dataset for Radiologic Visual Question Answering
  5. 2023Visual Question Answering in the Medical Domain
  6. 2025Adapting Lightweight Vision Language Models for Radiological Visual Question Answering
  7. 2024Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Ray
  8. 2025Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations