mlsanityData to 7 Oct 2026

Mask3D: Mask Transformer for 3D Semantic Instance Segmentation

arXiv 2210.03105 ↗First posted 6 Oct 2022cs.CVcited by 299

Not on the map. Why?

Cited by 40 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Modern 3D semantic instance segmentation approaches predominantly rely on specialized voting mechanisms followed by carefully designed geometric clustering techniques. Building on the successes of recent Transformer-based methods for object detection and image segmentation, we propose the first Transformer-based approach for 3D semantic instance segmentation. We show that we can leverage generic Transformer building blocks to directly predict instance masks from 3D point clouds. In our model called Mask3D each object instance is represented as an instance query. Using Transformer decoders, the instance queries are learned by iteratively attending to point cloud features at multiple scales. Combined with point features, the instance queries directly yield all instance masks in parallel. Mask3D has several advantages over current state-of-the-art approaches, since it neither relies on (1) voting schemes which require hand-selected geometric properties (such as centers) nor (2) geometric grouping mechanisms requiring manually-tuned hyper-parameters (e.g. radii) and (3) enables a loss that directly optimizes instance masks. Mask3D sets a new state-of-the-art on ScanNet test (+6.2 mAP), S3DIS 6-fold (+10.1 mAP), STPLS3D (+11.2 mAP) and ScanNet200 test (+12.4 mAP).

Key facts. Cited by 299 papers in the map (data through 7 Oct 2026). Builds on 53 of its 64 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

53 of 64 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  10. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  12. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
38 more
  1. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  2. 2020Point Transformercited by 1,462
  3. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  4. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  5. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  6. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  7. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  8. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  9. 2016Dynamic Filter Networkscited by 318
  10. 2020PointGroup: Dual-Set Point Grouping for 3D Instance Segmentationcited by 318
  11. 2020Conditional Convolutions for Instance Segmentationcited by 314
  12. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  13. 2016Volumetric and Multi-View CNNs for Object Classification on 3D Datacited by 240
  14. 2017Submanifold Sparse Convolutional Networkscited by 223
  15. 20183D-SIS: 3D Semantic Instance Segmentation of RGB-D Scanscited by 216
  16. 2021Group-Free 3D Object Detection via Transformerscited by 215
  17. 2020Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contextscited by 210
  18. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 196
  19. 20203D Object Detection with Pointformercited by 194
  20. 2020OccuSeg: Occupancy-aware 3D Instance Segmentationcited by 177
  21. 2017SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentationcited by 168
  22. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 166
  23. 2019Learning Object Bounding Boxes for 3D Instance Segmentation on Point Cloudscited by 156
  24. 2018GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloudcited by 145
  25. 2015End-to-end people detection in crowded scenescited by 140
  26. 20203D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentationcited by 140
  27. 2021Hierarchical Aggregation for 3D Instance Segmentationcited by 126
  28. 2021Mix3D: Out-of-Context Data Augmentation for 3D Scenescited by 126
  29. 2021Fast Point Transformercited by 103
  30. 2021Instance Segmentation in 3D Scenes using Semantic Superpoint Tree Networkscited by 98
  31. 2021FCAF3D: Fully Convolutional Anchor-Free 3D Object Detectioncited by 97
  32. 2019Associatively Segmenting Instances and Semantics in Point Cloudscited by 86
  33. 2018Learning to predict crisp boundariescited by 81
  34. 20193D Instance Segmentation via Multi-Task Metric Learningcited by 79
  35. 2020DyCo3D: Robust Instance Segmentation of 3D Point Clouds through Dynamic Convolutioncited by 74
  36. 2022STPLS3D: A Large-Scale Synthetic and Real Aerial Photogrammetry 3D Point Cloud Datasetcited by 52
  37. 2017O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysiscited by 47
  38. 2022MaskGroup: Hierarchical Point Grouping and Masking for 3D Instance Segmentationcited by 29

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 83; 2025: 115; 2026: 78.

299 papers.

Sort:

By area:

  1. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 387
  2. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  3. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  4. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  5. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  6. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  7. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  8. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  9. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  10. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  11. 2023OneFormer3D: One Transformer for Unified Point Cloud Segmentationcited by 95
  12. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
38 more
  1. 2023OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentationcited by 82
  2. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  3. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  4. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  5. 2024Point Cloud Mamba: Point Cloud Learning via State Space Modelcited by 66
  6. 2024SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Groundingcited by 66
  7. 2023Visual Programming for Zero-shot Open-Vocabulary 3D Visual Groundingcited by 64
  8. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64
  9. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  10. 2023Fully Sparse 3D Occupancy Predictioncited by 61
  11. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  12. 2024MaskClustering: View Consensus based Mask Graph Clustering for Open-Vocabulary 3D Instance Segmentationcited by 56
  13. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  14. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 50
  15. 2025Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigationcited by 50
  16. 2024Point-SAM: Promptable 3D Segmentation Model for Point Cloudscited by 49
  17. 2024ScanReason: Empowering 3D Visual Grounding with Reasoning Capabilitiescited by 47
  18. 2023SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentationcited by 44
  19. 2023PartSLIP++: Enhancing Low-Shot 3D Part Segmentation via Multi-View Instance Segmentation and Maximum Likelihood Estimationcited by 37
  20. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  21. 2024LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferencescited by 36
  22. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  23. 2024Search3D: Hierarchical Open-Vocabulary 3D Segmentationcited by 31
  24. 2024Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentationcited by 30
  25. 2024Find Any Part in 3Dcited by 30
  26. 2024PO3AD: Predicting Point Offsets toward Better 3D Point Cloud Anomaly Detectioncited by 30
  27. 2025N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Modelscited by 28
  28. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 27
  29. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26
  30. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  31. 2024SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Modelscited by 26
  32. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understandingcited by 26
  33. 2024ODIN: A Single Model for 2D and 3D Segmentationcited by 25
  34. 2024RefMask3D: Language-Guided Transformer for 3D Referring Segmentationcited by 25
  35. 2024S2O: Static to Openable Enhancement for Articulated 3D Objectscited by 22
  36. 2025FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interactioncited by 22
  37. 2025Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysiscited by 22
  38. 2024UniDet3D: Multi-dataset Indoor 3D Object Detectioncited by 20

50 most cited of 299 All 299 (JSON)