# MedVista3D: Vision-Language Modeling for Reducing Diagnostic Errors in 3D CT Disease Detection, Understanding and Reporting

arXiv: https://arxiv.org/abs/2509.03800 · HTML: https://mlsanity.com/paper/2509.03800/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 46 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Demystifying CLIP Data](https://mlsanity.com/paper/2309.16671/) (2023)
- [PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents](https://mlsanity.com/paper/2303.07240/) (2023)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [Unsupervised Medical Image Translation with Adversarial Diffusion Models](https://mlsanity.com/paper/2207.08208/) (2022)
- [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023)
- [Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning](https://mlsanity.com/paper/2210.06044/) (2022)
- [MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training in Radiology](https://mlsanity.com/paper/2301.02228/) (2023)
- [Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset](https://mlsanity.com/paper/2406.06512/) (2024)
- [STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training](https://mlsanity.com/paper/2304.06716/) (2023)
- [Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumes](https://mlsanity.com/paper/2002.04752/) (2020)
- [UniMiSS: Universal Medical Self-Supervised Learning via Breaking Dimensionality Barrier](https://mlsanity.com/paper/2112.09356/) (2021)
- [Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding](https://mlsanity.com/paper/2501.14548/) (2025)
- [RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis](https://mlsanity.com/paper/2404.16754/) (2024)
- [Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment](https://mlsanity.com/paper/2008.09884/) (2020)
- [CXR-LLAVA: a multimodal large language model for interpreting chest X-ray images](https://mlsanity.com/paper/2310.18341/) (2023)
- [Contrastive Localized Language-Image Pre-Training](https://mlsanity.com/paper/2410.02746/) (2024)
- [Artificial General Intelligence for Medical Imaging Analysis](https://mlsanity.com/paper/2306.05480/) (2023)
- [CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarios](https://mlsanity.com/paper/2404.15272/) (2024)
- [Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammography](https://mlsanity.com/paper/2409.18119/) (2024)
- [Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?](https://mlsanity.com/paper/2410.13523/) (2024)

## Built on by (most cited first, 4 of 4)

- [Self-Supervised Dynamical System Representations for Physiological Time-Series](https://mlsanity.com/paper/2512.00239/) (2025)
- [Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT](https://mlsanity.com/paper/2603.02026/) (2026)
- [ASAP: Advancing Medical Volumetric Representation Learning with Anatomy-aware Semantically-adaptive Pre-training](https://mlsanity.com/paper/2606.00602/) (2026)
- [Volumetric Radiology AI in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2608.20549/) (2026)

## Abstract (arXiv)

Radiologic diagnostic errors-under-reading errors, inattentional blindness, and communication failures-remain prevalent in clinical practice. These issues often stem from missed localized abnormalities, limited global context, and variability in report language. These challenges are amplified in 3D imaging, where clinicians must examine hundreds of slices per scan. Addressing them requires systems with precise localized detection, global volume-level reasoning, and semantically consistent natural language reporting. However, existing 3D vision-language models are unable to meet all three needs jointly, lacking local-global understanding for spatial reasoning and struggling with the variability and noise of uncurated radiology reports. We present MedVista3D, a multi-scale semantic-enriched vision-language pretraining framework for 3D CT analysis. To enable joint disease detection and holistic interpretation, MedVista3D performs local and global image-text alignment for fine-grained representation learning within full-volume context. To address report variability, we apply language model rewrites and introduce a Radiology Semantic Matching Bank for semantics-aware alignment. MedVista3D achieves state-of-the-art performance on zero-shot disease classification, report retrieval, and medical visual question answering, while transferring well to organ segmentation and prognosis prediction. Code and datasets will be released.