# FastSAM3D: An Efficient Segment Anything Model for 3D Volumetric Medical Images

arXiv: https://arxiv.org/abs/2403.09827 · HTML: https://mlsanity.com/paper/2403.09827/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 27 of 27)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [MetaFormer Is Actually What You Need for Vision](https://mlsanity.com/paper/2111.11418/) (2021)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Faster Segment Anything: Towards Lightweight SAM for Mobile Applications](https://mlsanity.com/paper/2306.14289/) (2023)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [Segment Anything Model for Medical Image Analysis: an Experimental Study](https://mlsanity.com/paper/2304.10517/) (2023)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [LongNet: Scaling Transformers to 1,000,000,000 Tokens](https://mlsanity.com/paper/2307.02486/) (2023)
- [SAM-Med2D](https://mlsanity.com/paper/2308.16184/) (2023)
- [Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning](https://mlsanity.com/paper/1710.04043/) (2017)
- [SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images](https://mlsanity.com/paper/2310.15161/) (2023)
- [Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions](https://mlsanity.com/paper/2401.03495/) (2024)
- [A Survey for Foundation Models in Autonomous Driving](https://mlsanity.com/paper/2402.01105/) (2024)
- [SAM3D: Segment Anything Model in Volumetric Medical Images](https://mlsanity.com/paper/2309.03493/) (2023)
- [TinySAM: Pushing the Envelope for Efficient Segment Anything Model](https://mlsanity.com/paper/2312.13789/) (2023)
- [A Survey on Deep Learning for Skin Lesion Segmentation](https://mlsanity.com/paper/2206.00356/) (2022)
- [RepViT-SAM: Towards Real-Time Segmenting Anything](https://mlsanity.com/paper/2312.05760/) (2023)
- [TransNuSeg: A Lightweight Multi-Task Transformer for Nuclei Segmentation](https://mlsanity.com/paper/2307.08051/) (2023)
- [UI-Net: Interactive Artificial Neural Networks for Iterative Image Segmentation Based on a User Model](https://mlsanity.com/paper/1709.03450/) (2017)
- [MoViT: Memorizing Vision Transformers for Medical Image Analysis](https://mlsanity.com/paper/2303.15553/) (2023)
- [Towards the Unification of Generative and Discriminative Visual Foundation Model: A Survey](https://mlsanity.com/paper/2312.10163/) (2023)

## Built on by (most cited first, 14 of 14)

- [FairMedFM: Fairness Benchmarking for Medical Imaging Foundation Models](https://mlsanity.com/paper/2407.00983/) (2024)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [ROS-SAM: High-Quality Interactive Segmentation for Remote Sensing Moving Object](https://mlsanity.com/paper/2503.12006/) (2025)
- [Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation](https://mlsanity.com/paper/2503.04639/) (2025)
- [Recent Advances in Medical Imaging Segmentation: A Survey](https://mlsanity.com/paper/2505.09274/) (2025)
- [SAM & SAM 2 in 3D Slicer: SegmentWithSAM Extension for Annotating Medical Images](https://mlsanity.com/paper/2408.15224/) (2024)
- [SAMSOD: Rethinking SAM Optimization for RGB-T Salient Object Detection](https://mlsanity.com/paper/2510.03689/) (2025)
- [Intelligent Control of Robotic X-ray Devices using a Language-promptable Digital Twin](https://mlsanity.com/paper/2412.08020/) (2024)
- [InfoSAM: Fine-Tuning the Segment Anything Model from An Information-Theoretic Perspective](https://mlsanity.com/paper/2505.21920/) (2025)
- [Memorizing SAM: 3D Medical Segment Anything Model with Memorizing Transformer](https://mlsanity.com/paper/2412.13908/) (2024)
- [BodyGPS: Anatomical Positioning System](https://mlsanity.com/paper/2505.07744/) (2025)
- [ESICA: A Scalable Framework for Text-Guided 3D Medical Image Segmentation](https://mlsanity.com/paper/2604.24876/) (2026)
- [Geometry-Consistent Endoscopic Representations for Image-Guided Navigation via Structured Foundation Model Adaptation](https://mlsanity.com/paper/2606.17340/) (2026)
- [Confidence-Aware Teacher-Student Distillation for 3D Medical Segmentation](https://mlsanity.com/paper/2609.23815/) (2026)

## Abstract (arXiv)

Segment anything models (SAMs) are gaining attention for their zero-shot generalization capability in segmenting objects of unseen classes and in unseen domains when properly prompted. Interactivity is a key strength of SAMs, allowing users to iteratively provide prompts that specify objects of interest to refine outputs. However, to realize the interactive use of SAMs for 3D medical imaging tasks, rapid inference times are necessary. High memory requirements and long processing delays remain constraints that hinder the adoption of SAMs for this purpose. Specifically, while 2D SAMs applied to 3D volumes contend with repetitive computation to process all slices independently, 3D SAMs suffer from an exponential increase in model parameters and FLOPS. To address these challenges, we present FastSAM3D which accelerates SAM inference to 8 milliseconds per 128*128*128 3D volumetric image on an NVIDIA A100 GPU. This speedup is accomplished through 1) a novel layer-wise progressive distillation scheme that enables knowledge transfer from a complex 12-layer ViT-B to a lightweight 6-layer ViT-Tiny variant encoder without training from scratch; and 2) a novel 3D sparse flash attention to replace vanilla attention operators, substantially reducing memory needs and improving parallelization. Experiments on three diverse datasets reveal that FastSAM3D achieves a remarkable speedup of 527.38x compared to 2D SAMs and 8.75x compared to 3D SAMs on the same volumes without significant performance decline. Thus, FastSAM3D opens the door for low-cost truly interactive SAM-based 3D medical imaging segmentation with commonly used GPU hardware. Code is available at https://github.com/arcadelab/FastSAM3D.