# INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosis

arXiv: https://arxiv.org/abs/2311.10798 · HTML: https://mlsanity.com/paper/2311.10798/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 66 references that are in the map.
- First posted on arXiv 17 Nov 2023.

## Builds on (most cited first, 12 of 12)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs](https://mlsanity.com/paper/1901.07042/) (2019)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [BEHRT: Transformer for Electronic Health Records](https://mlsanity.com/paper/1907.09538/) (2019)
- [TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays](https://mlsanity.com/paper/1801.04334/) (2018)
- [Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences](https://mlsanity.com/paper/2201.11838/) (2022)
- [Deep Multi-modal Fusion of Image and Non-image Data in Disease Diagnosis and Prognosis: A Review](https://mlsanity.com/paper/2203.15588/) (2022)
- [RadFusion: Benchmarking Performance and Fairness for Multimodal Pulmonary Embolism Detection from CT and EHR](https://mlsanity.com/paper/2111.11665/) (2021)
- [A new paradigm for accelerating clinical data science at Stanford Medicine](https://mlsanity.com/paper/2003.10534/) (2020)
- [Early Prediction of Alzheimer's Disease Dementia Based on Baseline Hippocampal MRI and 1-Year Follow-Up Cognitive Measures Using Deep Recurrent Neural Networks](https://mlsanity.com/paper/1901.01451/) (2019)

## Built on by (most cited first, 27 of 27)

- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [QoQ-Med: Building Multimodal Clinical Foundation Models with Domain-Aware GRPO Training](https://mlsanity.com/paper/2506.00711/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Comprehensive language-image pre-training for 3D medical image understanding](https://mlsanity.com/paper/2510.15042/) (2025)
- [T3D: Advancing 3D Medical Vision-Language Pre-training by Learning Multi-View Visual Consistency](https://mlsanity.com/paper/2312.01529/) (2023)
- [CLIMB: Data Foundations for Large Scale Multimodal Clinical Foundation Models](https://mlsanity.com/paper/2503.07667/) (2025)
- [Bridged Semantic Alignment for Zero-shot 3D Medical Image Diagnosis](https://mlsanity.com/paper/2501.03565/) (2025)
- [Exploring Multimodal Large Language Models for Radiology Report Error-checking](https://mlsanity.com/paper/2312.13103/) (2023)
- [MedPix 2.0: A Comprehensive Multimodal Biomedical Data set for Advanced AI Applications with Retrieval Augmented Generation and Knowledge Graphs](https://mlsanity.com/paper/2407.02994/) (2024)
- [From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine](https://mlsanity.com/paper/2502.09242/) (2025)
- [Fusion or Confusion? Multimodal Complexity Is Not All You Need](https://mlsanity.com/paper/2512.22991/) (2025)
- [Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning](https://mlsanity.com/paper/2606.24570/) (2026)
- [Beyond Medical Diagnostics: How Medical Multimodal Large Language Models Think in Space](https://mlsanity.com/paper/2603.13800/) (2026)
- [Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Tasks](https://mlsanity.com/paper/2604.04133/) (2026)
- [X-ray2CTPA: Leveraging Diffusion Models to Enhance Pulmonary Embolism Classification](https://mlsanity.com/paper/2406.16109/) (2024)
- [Abn-BLIP: Abnormality-aligned Bootstrapping Language-Image Pre-training for Pulmonary Embolism Diagnosis and Report Generation from CTPA](https://mlsanity.com/paper/2503.02034/) (2025)
- [MedTri: A Platform for Structured Medical Report Normalization to Enhance Vision-Language Pretraining](https://mlsanity.com/paper/2602.22143/) (2026)
- [Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development](https://mlsanity.com/paper/2603.27460/) (2026)
- [TrajGPT: Irregular Time-Series Representation Learning for Health Trajectory Analysis](https://mlsanity.com/paper/2410.02133/) (2024)
- [Astra: a generalizable report generation foundation model for 3D computed tomography](https://mlsanity.com/paper/2605.31437/) (2026)
- [ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-training](https://mlsanity.com/paper/2606.00602/) (2026)
- [RadSight: Towards Perceptually Reliable Multimodal Radiology Image Understanding](https://mlsanity.com/paper/2607.22293/) (2026)
- [When are radiology reports useful for training medical image classifiers?](https://mlsanity.com/paper/2510.24385/) (2025)
- [Medical Image Understanding Improves Survival Prediction via Visual Instruction Tuning](https://mlsanity.com/paper/2604.18250/) (2026)
- [Fusion is not one-size-fits-all: Cross-Modal Representation Alignment for Time-to-Event Modeling](https://mlsanity.com/paper/2606.15038/) (2026)
- [Efficient Multimodal Clinical Question Answering for Pulmonary Embolism Risk Assessment](https://mlsanity.com/paper/2606.22442/) (2026)
- [Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning](https://mlsanity.com/paper/2610.14770/) (2026)

## Abstract (arXiv)

Synthesizing information from multiple data sources plays a crucial role in the practice of modern medicine. Current applications of artificial intelligence in medicine often focus on single-modality data due to a lack of publicly available, multimodal medical datasets. To address this limitation, we introduce INSPECT, which contains de-identified longitudinal records from a large cohort of patients at risk for pulmonary embolism (PE), along with ground truth labels for multiple outcomes. INSPECT contains data from 19,402 patients, including CT images, radiology report impression sections, and structured electronic health record (EHR) data (i.e. demographics, diagnoses, procedures, vitals, and medications). Using INSPECT, we develop and release a benchmark for evaluating several baseline modeling approaches on a variety of important PE related tasks. We evaluate image-only, EHR-only, and multimodal fusion models. Trained models and the de-identified dataset are made available for non-commercial use under a data use agreement. To the best of our knowledge, INSPECT is the largest multimodal dataset integrating 3D medical imaging and EHR for reproducible methods evaluation and research.