mlsanityData to 7 Oct 2026

SAM3D: Zero-Shot 3D Object Detection via Segment Anything Model

arXiv 2306.02245 ↗First posted 4 Jun 2023cs.CV eess.IVcited by 40

Not on the map. Why?

Abstract (arXiv)

With the development of large language models, many remarkable linguistic systems like ChatGPT have thrived and achieved astonishing success on many tasks, showing the incredible power of foundation models. In the spirit of unleashing the capability of foundation models on vision tasks, the Segment Anything Model (SAM), a vision foundation model for image segmentation, has been proposed recently and presents strong zero-shot ability on many downstream 2D tasks. However, whether SAM can be adapted to 3D vision tasks has yet to be explored, especially 3D object detection. With this inspiration, we explore adapting the zero-shot ability of SAM to 3D object detection in this paper. We propose a SAM-powered BEV processing pipeline to detect objects and get promising results on the large-scale Waymo open dataset. As an early attempt, our method takes a step toward 3D object detection with vision foundation models and presents the opportunity to unleash their power on 3D vision tasks. The code is released at https://github.com/DYZhang09/SAM3D.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 4 Jun 2023.

Builds on

28 of 37 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023Segment Anythingcited by 8,158
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  10. 2018PointPillars: Fast Encoders for Object Detection from Point Cloudscited by 1,707
  11. 2017VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detectioncited by 1,323
  12. 2020Center-based 3D Object Detection and Trackingcited by 1,143
16 more
  1. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  2. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  3. 2020Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detectioncited by 409
  4. 2021PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detectioncited by 232
  5. 2023VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Trackingcited by 221
  6. 2023Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imagingcited by 161
  7. 2023Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applicationscited by 121
  8. 2022CenterFormer: Center-based Transformer for 3D Object Detectioncited by 99
  9. 2023Can SAM Segment Anything? When SAM Meets Camouflaged Object Detectioncited by 99
  10. 2023Anything-3D: Towards Single-view Anything Reconstruction in the Wildcited by 91
  11. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  12. 2023SAM Struggles in Concealed Scenes -- Empirical Study on Segment Anythingcited by 75
  13. 2023Segment Anything in 3D with Radiance Fieldscited by 61
  14. 2023Can SAM Segment Polyps?cited by 59
  15. 2023CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Modelcited by 47
  16. 2023DDS3D: Dense Pseudo-Labels with Dynamic Threshold for Semi-Supervised 3D Object Detectioncited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 20; 2025: 11; 2026: 2.

40 papers.

Sort:

By area:

  1. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
  2. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  3. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  4. 2023SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimationcited by 115
  5. 2024A Survey for Foundation Models in Autonomous Drivingcited by 51
  6. 2023SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentationcited by 44
  7. 2024SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decompositioncited by 29
  8. 2024Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunitiescited by 24
  9. 2024Fantastic Animals and Where to Find Them: Segment Any Marine Animal with Dual SAMcited by 23
  10. 2024RoboFusion: Towards Robust Multi-Modal 3D Object Detection via SAMcited by 20
  11. 2024A Unified Framework for 3D Scene Understandingcited by 15
  12. 2024iSeg: Interactive 3D Segmentation via Interactive Attentioncited by 14
28 more
  1. 2025AutoOcc: Automatic Open-Ended Semantic Occupancy Annotation via Vision-Language Guided Gaussian Splattingcited by 14
  2. 2024MixSup: Mixed-grained Supervision for Label-efficient LiDAR-based 3D Object Detectioncited by 9
  3. 2024Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detectioncited by 9
  4. 2024Make Your ViT-based Multi-view 3D Detectors Faster via Token Compressioncited by 9
  5. 2025LeAP: Consistent multi-domain 3D labeling using Foundation Modelscited by 8
  6. 2024ZOPP: A Framework of Zero-shot Offboard Panoptic Perception for Autonomous Drivingcited by 6
  7. 2025SP3D: Boosting Sparsely-Supervised 3D Object Detection via Accurate Cross-Modal Semantic Promptscited by 6
  8. 2025Foundation Models for Autonomous Driving Perception: A Survey Through Core Capabilitiescited by 6
  9. 2023Diffusion-based 3D Object Detection with Random Boxescited by 5
  10. 2023Segment Any Events via Weighted Adaptation of Pivotal Tokenscited by 5
  11. 2024On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Surveycited by 5
  12. 2023Industrial Segment Anything -- a Case Study in Aircraft Manufacturing, Intralogistics, Maintenance, Repair, and Overhaulcited by 4
  13. 2024EchoONE: Segmenting Multiple echocardiography Planes in One Modelcited by 4
  14. 2025SALT: A Flexible Semi-Automatic Labeling Tool for General LiDAR Point Clouds with Cross-Scene Adaptability and 4D Consistencycited by 4
  15. 2025EA3D: Online Open-World 3D Object Extraction from Streaming Videoscited by 4
  16. 2025Zoo3D: Zero-Shot 3D Object Detection at Scene Levelcited by 4
  17. 2024SamLP: A Customized Segment Anything Model for License Plate Detectioncited by 3
  18. 2024You Only Look Bottom-Up for Monocular 3D Object Detectioncited by 3
  19. 2024FM-Fusion: Instance-aware Semantic Mapping Boosted by Vision-Language Foundation Modelscited by 3
  20. 2024A Training-Free Framework for Video License Plate Tracking and Recognition with Only One-Shotcited by 2
  21. 2025SERES: Semantic-aware neural reconstruction from sparse viewscited by 1
  22. 2026PartRAG: Retrieval-Augmented Part-Level 3D Generation and Editingcited by 1
  23. 2024ELiTe: Efficient Image-to-LiDAR Knowledge Transfer for Semantic Segmentationcited by 0
  24. 2024Test-time Correction: An Online 3D Detection System via Visual Promptingcited by 0
  25. 2025AutoExpert: Automating 3D LiDAR Annotation from Expert-Crafted Guidelinescited by 0
  26. 2025FocalClick-XL: Towards Unified and High-quality Interactive Segmentationcited by 0
  27. 2025Segmentation-Driven Initialization for Sparse-view 3D Gaussian Splattingcited by 0
  28. 2026Towards robust multimodal 3D object detection via visual foundation modelscited by 0