mlsanityData to 14 Oct 2026

INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosis

arXiv 2311.10798 ↗First posted 17 Nov 2023cs.LG cs.AI cs.CVcited by 27

Not on the map. Why?

Abstract (arXiv)

Synthesizing information from multiple data sources plays a crucial role in the practice of modern medicine. Current applications of artificial intelligence in medicine often focus on single-modality data due to a lack of publicly available, multimodal medical datasets. To address this limitation, we introduce INSPECT, which contains de-identified longitudinal records from a large cohort of patients at risk for pulmonary embolism (PE), along with ground truth labels for multiple outcomes. INSPECT contains data from 19,402 patients, including CT images, radiology report impression sections, and structured electronic health record (EHR) data (i.e. demographics, diagnoses, procedures, vitals, and medications). Using INSPECT, we develop and release a benchmark for evaluating several baseline modeling approaches on a variety of important PE related tasks. We evaluate image-only, EHR-only, and multimodal fusion models. Trained models and the de-identified dataset are made available for non-commercial use under a data use agreement. To the best of our knowledge, INSPECT is the largest multimodal dataset integrating 3D medical imaging and EHR for reproducible methods evaluation and research.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 12 of its 66 references that are in the map. First posted on arXiv 17 Nov 2023.

Builds on

12 of 66 references.

References · 12

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  3. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  4. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  5. 2018Scalable and accurate deep learning for electronic health recordscited by 280
  6. 2019BEHRT: Transformer for Electronic Health Recordscited by 263
  7. 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
  8. 2022Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequencescited by 69
  9. 2022Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Reviewcited by 37
  10. 2021RadFusion: Benchmarking Performance and Fairness for Multimodal Pulmonary Embolism Detection from CT and EHRcited by 32
  11. 2020A new paradigm for accelerating clinical data science at Stanford Medicinecited by 17
  12. 2019Early Prediction of Alzheimer's Disease Dementia Based on Baseline Hippocampal MRI and 1-Year Follow-Up Cognitive Measures Using Deep Recurrent Neural Networkscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 9; 2026: 12.

27 papers.

Sort:

By area:

  1. 2023CLIP in Medical Imaging: A Surveycited by 40
  2. 2025QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Trainingcited by 38
  3. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  4. 2025Comprehensive language-image pre-training for 3D medical image understandingcited by 22
  5. 2023T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistencycited by 14
  6. 2025CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Modelscited by 13
  7. 2025Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosiscited by 9
  8. 2023Exploring Multimodal Large Language Models for Radiology Report Error-checkingcited by 6
  9. 2024MedPix 2.0: A Comprehensive Multimodal Biomedical Data set for Advanced AI Applications with Retrieval Augmented Generation and Knowledge Graphscited by 6
  10. 2025From large language models to multimodal AI: A scoping review on the potential of generative AI in medicinecited by 5
  11. 2025Fusion or Confusion? Multimodal Complexity Is Not All You Needcited by 5
  12. 2026Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learningcited by 4
15 more
  1. 2026Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Spacecited by 3
  2. 2026Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Taskscited by 3
  3. 2024X-ray2CTPA: Leveraging Diffusion Models to Enhance Pulmonary Embolism Classificationcited by 2
  4. 2025Abn-BLIP: Abnormality-aligned Bootstrapping Language-Image Pre-training for Pulmonary Embolism Diagnosis and Report Generation from CTPAcited by 2
  5. 2026MedTri: A Platform for Structured Medical Report Normalization to Enhance Vision-Language Pretrainingcited by 2
  6. 2026Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Developmentcited by 2
  7. 2024TrajGPT: Irregular Time-Series Representation Learning for Health Trajectory Analysiscited by 1
  8. 2026Astra: a generalizable report generation foundation model for 3D computed tomographycited by 1
  9. 2026ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-trainingcited by 1
  10. 2026RadSight: Towards Perceptually Reliable Multimodal Radiology Image Understandingcited by 1
  11. 2025When are radiology reports useful for training medical image classifiers?cited by 0
  12. 2026Medical Image Understanding Improves Survival Prediction via Visual Instruction Tuningcited by 0
  13. 2026Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modelingcited by 0
  14. 2026Efficient Multimodal Clinical Question Answering for Pulmonary Embolism Risk Assessmentcited by 0
  15. 2026Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learningcited by 0