# VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging

arXiv: https://arxiv.org/abs/2406.05285 · HTML: https://mlsanity.com/paper/2406.05285/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 61 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 28 of 28)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [UNETR: Transformers for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.10504/) (2021)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation](https://mlsanity.com/paper/2304.12620/) (2023)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [3D MRI brain tumor segmentation using autoencoder regularization](https://mlsanity.com/paper/1810.11654/) (2018)
- [AbdomenCT-1K: Is Abdominal Organ Segmentation A Solved Problem?](https://mlsanity.com/paper/2010.14808/) (2020)
- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation](https://mlsanity.com/paper/1506.06448/) (2015)
- [VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images](https://mlsanity.com/paper/2001.09193/) (2020)
- [FocalClick: Towards Practical Interactive Image Segmentation](https://mlsanity.com/paper/2204.02574/) (2022)
- [SegVol: Universal and Interactive Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2311.13385/) (2023)
- [Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challenge](https://mlsanity.com/paper/2308.05862/) (2023)
- [DiNTS: Differentiable Neural Network Topology Search for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.15954/) (2021)
- [One-Prompt to Segment All Medical Images](https://mlsanity.com/paper/2305.10300/) (2023)
- [SAM3D: Segment Anything Model in Volumetric Medical Images](https://mlsanity.com/paper/2309.03493/) (2023)
- [Continual Segment: Towards a Single, Unified and Accessible Continual Segmentation Model of 143 Whole-body Organs in CT Scans](https://mlsanity.com/paper/2302.00162/) (2023)
- [Automated 3D Segmentation of Kidneys and Tumors in MICCAI KiTS 2023 Challenge](https://mlsanity.com/paper/2310.04110/) (2023)
- [AeroPath: An airway segmentation benchmark dataset with challenging pathology](https://mlsanity.com/paper/2311.01138/) (2023)
- [Multi-Center Fetal Brain Tissue Annotation (FeTA) Challenge 2022 Results](https://mlsanity.com/paper/2402.09463/) (2024)
- [Aorta Segmentation from 3D CT in MICCAI SEG.A. 2023 Challenge](https://mlsanity.com/paper/2310.04114/) (2023)
- [SegICL: A Multimodal In-context Learning Framework for Enhanced Segmentation in Medical Imaging](https://mlsanity.com/paper/2403.16578/) (2024)

## Built on by (most cited first, 30 of 64)

- [MedSAM2: Segment Anything in 3D Medical Images and Videos](https://mlsanity.com/paper/2504.03600/) (2025)
- [Scaling Self-Supervised and Cross-Modal Pretraining for Volumetric CT Transformers](https://mlsanity.com/paper/2511.17209/) (2025)
- [A Short Review and Evaluation of SAM2's Performance in 3D CT Image Segmentation](https://mlsanity.com/paper/2408.11210/) (2024)
- [Evaluating Cell AI Foundation Models in Kidney Pathology with Human-in-the-Loop Enrichment](https://mlsanity.com/paper/2411.00078/) (2024)
- [Conformal Lesion Segmentation for 3D Medical Images](https://mlsanity.com/paper/2510.17897/) (2025)
- [MedNeXt-v2: Scaling 3D ConvNeXts for Large-Scale Supervised Representation Learning in Medical Image Segmentation](https://mlsanity.com/paper/2512.17774/) (2025)
- [3DMedAgent: Unified Perception-to-Understanding for 3D Medical Analysis](https://mlsanity.com/paper/2602.18064/) (2026)
- [MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflows](https://mlsanity.com/paper/2603.24649/) (2026)
- [OmniMRI: A Unified Vision--Language Foundation Model for Generalist MRI Interpretation](https://mlsanity.com/paper/2508.17524/) (2025)
- [Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directions](https://mlsanity.com/paper/2511.01284/) (2025)
- [See More, Change Less: Anatomy-Aware Diffusion for Contrast Enhancement](https://mlsanity.com/paper/2512.07251/) (2025)
- [ENSAM: an efficient foundation model for interactive segmentation of 3D medical images](https://mlsanity.com/paper/2509.15874/) (2025)
- [Dynamic Prompt Generation for Interactive 3D Medical Image Segmentation Training](https://mlsanity.com/paper/2510.03189/) (2025)
- [TAP-CT: 3D Task-Agnostic Pretraining of Computed Tomography Foundation Models](https://mlsanity.com/paper/2512.00872/) (2025)
- [Foundation Models in Biomedical Imaging: Turning Hype into Reality](https://mlsanity.com/paper/2512.15808/) (2025)
- [NodMAISI: Nodule-Oriented Medical AI for Synthetic Imaging](https://mlsanity.com/paper/2512.18038/) (2025)
- [Revisiting foundation models for cell instance segmentation](https://mlsanity.com/paper/2603.17845/) (2026)
- [Towards Autonomous and Auditable Medical Imaging Model Development](https://mlsanity.com/paper/2607.10522/) (2026)
- [Virtual Patients, Real Gains: Digital Twin-Based Simulated CT for Multitask Lung Nodule Analysis](https://mlsanity.com/paper/2502.21187/) (2025)
- [Foundation Models for Zero-Shot Segmentation of Scientific Images without AI-Ready Data](https://mlsanity.com/paper/2506.24039/) (2025)
- [TomoGraphView: 3D Medical Image Classification with Omnidirectional Slice Representations and Graph Neural Networks](https://mlsanity.com/paper/2511.09605/) (2025)
- [Medal S: Spatio-Textual Prompt Model for Medical Segmentation](https://mlsanity.com/paper/2511.13001/) (2025)
- [Tumor-anchored deep feature random forests for out-of-distribution detection in lung cancer segmentation](https://mlsanity.com/paper/2512.08216/) (2025)
- [Universality Reconsidered: Rethinking the Validation of Foundation Models for General-Purpose 3D Medical Segmentation](https://mlsanity.com/paper/2602.07643/) (2026)
- [A Unified Framework for Joint Detection of Lacunes and Enlarged Perivascular Spaces](https://mlsanity.com/paper/2603.04243/) (2026)
- [Developing Foundation Models for Universal Segmentation from 3D Whole-Body Positron Emission Tomography](https://mlsanity.com/paper/2603.11627/) (2026)
- [VoxCor: Training-Free Volumetric Features for Multimodal Voxel Correspondence](https://mlsanity.com/paper/2605.13798/) (2026)
- [Foundation VAEs for 3D CT Reconstruction, Augmentation, and Generation](https://mlsanity.com/paper/2605.30893/) (2026)
- [Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models](https://mlsanity.com/paper/2607.04199/) (2026)
- [Anatomy Contextualized Adaptation of CT Foundation Models](https://mlsanity.com/paper/2607.27154/) (2026)

## Abstract (arXiv)

Foundation models for interactive segmentation in 2D natural images and videos have sparked significant interest in building 3D foundation models for medical imaging. However, the domain gaps and clinical use cases for 3D medical imaging require a dedicated model that diverges from existing 2D solutions. Specifically, such foundation models should support a full workflow that can actually reduce human effort. Treating 3D medical images as sequences of 2D slices and reusing interactive 2D foundation models seems straightforward, but 2D annotation is too time-consuming for 3D tasks. Moreover, for large cohort analysis, it's the highly accurate automatic segmentation models that reduce the most human effort. However, these models lack support for interactive corrections and lack zero-shot ability for novel structures, which is a key feature of "foundation". While reusing pre-trained 2D backbones in 3D enhances zero-shot potential, their performance on complex 3D structures still lags behind leading 3D models. To address these issues, we present VISTA3D, Versatile Imaging SegmenTation and Annotation model, that targets to solve all these challenges and requirements with one unified foundation model. VISTA3D is built on top of the well-established 3D segmentation pipeline, and it is the first model to achieve state-of-the-art performance in both 3D automatic (supporting 127 classes) and 3D interactive segmentation, even when compared with top 3D expert models on large and diverse benchmarks. Additionally, VISTA3D's 3D interactive design allows efficient human correction, and a novel 3D supervoxel method that distills 2D pretrained backbones grants VISTA3D top 3D zero-shot performance. We believe the model, recipe, and insights represent a promising step towards a clinically useful 3D foundation model. Code and weights are publicly available at https://github.com/Project-MONAI/VISTA.