# SAM3D: Segment Anything in 3D Scenes arXiv: https://arxiv.org/abs/2306.03908 ยท HTML: https://mlsanity.com/paper/2306.03908/ ## Key facts - Cited by 157 papers in the map (data through 14 Oct 2026). - Builds on 14 of its 16 references that are in the map. - First posted on arXiv 6 Jun 2023. ## Builds on (most cited first, 14 of 14) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023) - [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016) - [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022) - [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017) - [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022) - [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018) - [Point Transformer V2: Grouped Vector Attention and Partition-based Pooling](https://mlsanity.com/paper/2210.05666/) (2022) - [3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation](https://mlsanity.com/paper/1803.10409/) (2018) - [Virtual Multi-view Fusion for 3D Semantic Segmentation](https://mlsanity.com/paper/2007.13138/) (2020) - [Anything-3D: Towards Single-view Anything Reconstruction in the Wild](https://mlsanity.com/paper/2304.10261/) (2023) - [Multi-view PointNet for 3D Scene Understanding](https://mlsanity.com/paper/1909.13603/) (2019) - [SAD: Segment Any RGBD](https://mlsanity.com/paper/2305.14207/) (2023) ## Built on by (most cited first, 30 of 157) - [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023) - [Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance](https://mlsanity.com/paper/2312.10671/) (2023) - [SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation](https://mlsanity.com/paper/2311.15707/) (2023) - [GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models](https://mlsanity.com/paper/2501.01428/) (2025) - [SAMPart3D: Segment Any Part in 3D Objects](https://mlsanity.com/paper/2411.07184/) (2024) - [PARTFIELD: Learning 3D Feature Fields for Part Segmentation and Beyond](https://mlsanity.com/paper/2504.11451/) (2025) - [SAI3D: Segment Any Instance in 3D Scenes](https://mlsanity.com/paper/2312.11557/) (2023) - [OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation](https://mlsanity.com/paper/2309.00616/) (2023) - [Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels](https://mlsanity.com/paper/2312.17232/) (2023) - [OmniPart: Part-Aware 3D Generation with Semantic Decoupling and Structural Cohesion](https://mlsanity.com/paper/2507.06165/) (2025) - [MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2401.07745/) (2024) - [Part123: Part-aware 3D Reconstruction from a Single-view Image](https://mlsanity.com/paper/2405.16888/) (2024) - [Point-SAM: Promptable 3D Segmentation Model for Point Clouds](https://mlsanity.com/paper/2406.17741/) (2024) - [SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2311.17707/) (2023) - [OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation](https://mlsanity.com/paper/2403.14418/) (2024) - [P3-SAM: Native 3D Part Segmentation](https://mlsanity.com/paper/2509.06784/) (2025) - [EmbodiedSAM: Online Segment Any 3D Thing in Real Time](https://mlsanity.com/paper/2408.11811/) (2024) - [SAM-guided Graph Cut for 3D Instance Segmentation](https://mlsanity.com/paper/2312.08372/) (2023) - [Search3D: Hierarchical Open-Vocabulary 3D Segmentation](https://mlsanity.com/paper/2409.18431/) (2024) - [GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understanding](https://mlsanity.com/paper/2403.09639/) (2024) - [ObjectGS: Object-aware Scene Reconstruction and Scene Understanding via Gaussian Splatting](https://mlsanity.com/paper/2507.15454/) (2025) - [Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities](https://mlsanity.com/paper/2401.08045/) (2024) - [SlimSAM: 0.1% Data Makes Segment Anything Slim](https://mlsanity.com/paper/2312.05284/) (2023) - [OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splatting](https://mlsanity.com/paper/2506.07697/) (2025) - [ZeroPS: High-quality Cross-modal Knowledge Transfer for Zero-Shot 3D Part Segmentation](https://mlsanity.com/paper/2311.14262/) (2023) - [PartSTAD: 2D-to-3D Part Segmentation Task Adaptation](https://mlsanity.com/paper/2401.05906/) (2024) - [SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Manners](https://mlsanity.com/paper/2408.16768/) (2024) - [Visual Foundation Models Boost Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2403.10001/) (2024) - [A Unified Framework for 3D Scene Understanding](https://mlsanity.com/paper/2407.03263/) (2024) - [Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Tracking](https://mlsanity.com/paper/2411.16183/) (2024) ## Abstract (arXiv) In this work, we propose SAM3D, a novel framework that is able to predict masks in 3D point clouds by leveraging the Segment-Anything Model (SAM) in RGB images without further training or finetuning. For a point cloud of a 3D scene with posed RGB images, we first predict segmentation masks of RGB images with SAM, and then project the 2D masks into the 3D points. Later, we merge the 3D masks iteratively with a bottom-up merging approach. At each step, we merge the point cloud masks of two adjacent frames with the bidirectional merging approach. In this way, the 3D masks predicted from different frames are gradually merged into the 3D masks of the whole 3D scene. Finally, we can optionally ensemble the result from our SAM3D with the over-segmentation results based on the geometric information of the 3D scenes. Our approach is experimented with ScanNet dataset and qualitative results demonstrate that our SAM3D achieves reasonable and fine-grained 3D segmentation results without any training or finetuning of SAM.