mlsanityData to 7 Oct 2026

EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images

arXiv 2310.18652 ↗First posted 28 Oct 2023cs.CL cs.AI cs.CVcited by 53

Not on the map. Why?

Abstract (arXiv)

Electronic Health Records (EHRs), which contain patients' medical histories in various multi-modal formats, often overlook the potential for joint reasoning across imaging and table modalities underexplored in current EHR Question Answering (QA) systems. In this paper, we introduce EHRXQA, a novel multi-modal question answering dataset combining structured EHRs and chest X-ray images. To develop our dataset, we first construct two uni-modal resources: 1) The MIMIC-CXR-VQA dataset, our newly created medical visual question answering (VQA) benchmark, specifically designed to augment the imaging modality in EHR QA, and 2) EHRSQL (MIMIC-IV), a refashioned version of a previously established table-based EHR QA dataset. By integrating these two uni-modal resources, we successfully construct a multi-modal EHR QA dataset that necessitates both uni-modal and cross-modal reasoning. To address the unique challenges of multi-modal questions within EHRs, we propose a NeuralSQL-based strategy equipped with an external VQA API. This pioneering endeavor enhances engagement with multi-modal EHR sources and we believe that our dataset can catalyze advances in real-world medical scenarios such as clinical decision-making and research. EHRXQA is available at https://github.com/baeseongsu/ehrxqa.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 33 of its 62 references that are in the map. First posted on arXiv 28 Oct 2023.

Builds on

33 of 62 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  7. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  8. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  9. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  10. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  11. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  12. 2022Binding Language Models in Symbolic Languagescited by 224
21 more
  1. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  2. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  3. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 134
  4. 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 120
  5. 2021WebQA: Multihop and Multimodal QAcited by 109
  6. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  7. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96
  8. 2021Medical Visual Question Answering: A Surveycited by 93
  9. 2021Chest ImaGenome Dataset for Clinical Reasoningcited by 89
  10. 2019Text-to-SQL Generation for Question Answering on Electronic Medical Recordscited by 76
  11. 2020VQA-LOL: Visual Question Answering under the Lens of Logiccited by 48
  12. 2023EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Recordscited by 42
  13. 2020ManyModalQA: Modality Disambiguation and QA over Diverse Inputscited by 38
  14. 2022Conversational Question Answering on Heterogeneous Sourcescited by 26
  15. 2022DrugEHRQA: A Question Answering Dataset on Structured and Unstructured Electronic Health Records For Medicine Related Queriescited by 17
  16. 2022Learning to Ask Like a Physiciancited by 16
  17. 2023Interpretable Medical Image Visual Question Answering via Multi-Modal Relationship Graph Learningcited by 16
  18. 2021Question Answering for Complex Electronic Health Records Database using Unified Encoder-Decoder Architecturecited by 9
  19. 2023Towards Understanding the Generalization of Medical Text-to-SQL Models and Datasetscited by 9
  20. 2023LeafAI: query generator for clinical cohort discovery rivaling a human programmercited by 8
  21. 2023Towards Multi-Modal DBMSs for Seamless Querying of Texts and Tablescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 17; 2026: 21.

53 papers.

Sort:

By area:

  1. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 162
  2. 2023CLIP in Medical Imaging: A Surveycited by 40
  3. 2025Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generationcited by 39
  4. 2025MedicalAgentsBench for Complex Medical Reasoning: Comparing Internalized Reasoning Models versus Externalized Agent-based Frameworkscited by 38
  5. 2024EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summariescited by 33
  6. 2024Libra: Leveraging Temporal Images for Biomedical Radiology Analysiscited by 29
  7. 2024GEMeX: A Large-Scale, Groundable, and Explainable Medical VQA Benchmark for Chest X-ray Diagnosiscited by 28
  8. 2025ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understandingcited by 20
  9. 2024Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlookscited by 18
  10. 2024M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretationcited by 16
  11. 2024SM3-Text-to-Query: Synthetic Multi-Model Medical Text-to-Query Benchmarkcited by 10
  12. 2025MAP: Evaluation and Multi-Agent Enhancement of Large Language Models for Inpatient Pathwayscited by 9
38 more
  1. 2025MedOrch: Medical Diagnosis with Tool-Augmented Reasoning Agents for Flexible Extensibilitycited by 9
  2. 2024Enhancing Human-Computer Interaction in Chest X-ray Analysis using Vision and Language Model with Eye Gaze Patternscited by 8
  3. 2025AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretationcited by 8
  4. 2025Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoningcited by 8
  5. 2024Memory-Augmented Multimodal LLMs for Surgical VQA via Self-Contained Inquirycited by 6
  6. 2025SilVar-Med: A Speech-Driven Visual Language Model for Explainable Abnormality Detection in Medical Imagingcited by 5
  7. 2025MOTOR: Multimodal Optimal Transport via Grounded Retrieval in Medical Visual Question Answeringcited by 5
  8. 2026Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigationcited by 5
  9. 2026A Reasoning-Enabled Vision-Language Foundation Model for Chest X-ray Interpretationcited by 4
  10. 2025VinDr-CXR-VQA: A Visual Question Answering Dataset for Explainable Chest X-Ray Analysis with Multi-Task Learningcited by 3
  11. 2026VT-Bench: A Unified Benchmark for Visual-Tabular Multi-Modal Learningcited by 3
  12. 2026AgentRx: A Benchmark Study of LLM Agents for Multimodal Clinical Prediction Taskscited by 3
  13. 2024WoLF: Wide-scope Large Language Model Framework for CXR Understandingcited by 2
  14. 2024Towards Predicting Temporal Changes in a Patient's Chest X-ray Images based on Electronic Health Recordscited by 2
  15. 2025Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classificationcited by 2
  16. 2026Probing the Reliability of Driving VLMs: From Inconsistent Responses to Grounded Temporal Reasoningcited by 2
  17. 2026CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agentscited by 2
  18. 2024Evaluating LLMs for Text-to-SQL Generation With Complex SQL Workloadcited by 1
  19. 2024Expert-level vision-language foundation model for real-world radiology and comprehensive evaluationcited by 1
  20. 2024SURE-VQA: Systematic Understanding of Robustness Evaluation in Medical VQA Taskscited by 1
  21. 2025Enhancing Multi-task Learning Capability of Medical Generalist Foundation Model via Image-centric Multi-annotation Datacited by 1
  22. 2025Barriers in Integrating Medical Visual Question Answering into Radiology Workflows: A Scoping Review and Clinicians' Insightscited by 1
  23. 2025A Multi-Agent System for Complex Reasoning in Radiology Visual Question Answeringcited by 1
  24. 2025UCAgents: Unidirectional Convergence for Visual Evidence Anchored Multi-Agent Medical Decision-Makingcited by 1
  25. 2026EHRNavigator: A Multi-Agent System for Patient-Level Clinical Question Answering over Heterogeneous Electronic Health Recordscited by 1
  26. 2026Med-MMFL: A Multimodal Federated Learning Benchmark in Healthcarecited by 1
  27. 2026MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rayscited by 1
  28. 2026Mind the Tool Failures: Achieving Synergistic Tool Gains for Medical Agentscited by 1
  29. 2025Play by the Type Rules: Inferring Constraints for LLM Functions in Declarative Programscited by 0
  30. 2025CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistencycited by 0
  31. 2026HyperWalker: Dynamic Hypergraph-Based Deep Diagnosis for Multi-Hop Clinical Modeling across EHR and X-Ray in Medical VLMscited by 0
  32. 2026MMRad-22K: A Structured Multimodal Evidence Dataset for Chest X-ray Report Generationcited by 0
  33. 2026MuteBench: Modality Unavailability Tolerance Evaluation for Incomplete Multimodal Fusioncited by 0
  34. 2026UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretationcited by 0
  35. 2026Teach-to-Reason: Competition-Guided Reasoning with a Self-Improving Teachercited by 0
  36. 2026M3 QuestionIng: Multi-modal Multi-span Medical Question Answeringcited by 0
  37. 2026CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretationcited by 0
  38. 2026MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domaincited by 0

50 most cited of 53 All 53 (JSON)