# Deep Learning Based 3D Segmentation: A Survey

arXiv: https://arxiv.org/abs/2103.05423 · HTML: https://mlsanity.com/paper/2103.05423/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 117 of its 258 references that are in the map.
- First posted on arXiv 9 Mar 2021.

## Builds on (most cited first, 30 of 117)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Deep Learning for 3D Point Clouds: A Survey](https://mlsanity.com/paper/1912.12033/) (2019)
- [DeepGCNs: Can GCNs Go as Deep as CNNs?](https://mlsanity.com/paper/1904.03751/) (2019)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [Point-Voxel CNN for Efficient 3D Deep Learning](https://mlsanity.com/paper/1907.03739/) (2019)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs](https://mlsanity.com/paper/1704.02901/) (2017)
- [PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation](https://mlsanity.com/paper/2003.14032/) (2020)
- [SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters](https://mlsanity.com/paper/1803.11527/) (2018)
- [PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling](https://mlsanity.com/paper/2003.00492/) (2020)
- [SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud](https://mlsanity.com/paper/1710.07368/) (2017)

## Built on by (most cited first, 30 of 30)

- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [Vision Language Models in Autonomous Driving: A Survey and Outlook](https://mlsanity.com/paper/2310.14414/) (2023)
- [NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes](https://mlsanity.com/paper/2111.13260/) (2021)
- [Grounding 3D Object Affordance from 2D Interactions in Images](https://mlsanity.com/paper/2303.10437/) (2023)
- [CloSe: A 3D Clothing Segmentation Dataset and Model](https://mlsanity.com/paper/2401.12051/) (2024)
- [3D Vision with Transformers: A Survey](https://mlsanity.com/paper/2208.04309/) (2022)
- [Scalable 3D Panoptic Segmentation As Superpoint Graph Clustering](https://mlsanity.com/paper/2401.06704/) (2024)
- [NeuralBF: Neural Bilateral Filtering for Top-down Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2207.09978/) (2022)
- [Three-dimensional microstructure generation using generative adversarial neural networks in the context of continuum micromechanics](https://mlsanity.com/paper/2206.01693/) (2022)
- [Leveraging VLM-Based Pipelines to Annotate 3D Objects](https://mlsanity.com/paper/2311.17851/) (2023)
- [DA Wand: Distortion-Aware Selection using Neural Mesh Parameterization](https://mlsanity.com/paper/2212.06344/) (2022)
- [A Review of Panoptic Segmentation for Mobile Mapping Point Clouds](https://mlsanity.com/paper/2304.13980/) (2023)
- [SARNet: Semantic Augmented Registration of Large-Scale Urban Point Clouds](https://mlsanity.com/paper/2206.13117/) (2022)
- [On the Calibration of Uncertainty Estimation in LiDAR-based Semantic Segmentation](https://mlsanity.com/paper/2308.02248/) (2023)
- [HAL3D: Hierarchical Active Learning for Fine-Grained 3D Part Labeling](https://mlsanity.com/paper/2301.10460/) (2023)
- [MeT: A Graph Transformer for Semantic Segmentation of 3D Meshes](https://mlsanity.com/paper/2307.01115/) (2023)
- [Self-Supervised Contrastive Representation Learning for 3D Mesh Segmentation](https://mlsanity.com/paper/2208.04278/) (2022)
- [Semantic Segmentation-Assisted Instance Feature Fusion for Multi-Level 3D Part Instance Segmentation](https://mlsanity.com/paper/2208.04766/) (2022)
- [Learning 3D Semantics from Pose-Noisy 2D Images with Hierarchical Full Attention Network](https://mlsanity.com/paper/2204.08084/) (2022)
- [On the calibration of underrepresented classes in LiDAR-based semantic segmentation](https://mlsanity.com/paper/2210.06811/) (2022)
- [Cross-View Urban Traffic Dataset: Drone-Supervised Ground Truth for Monocular Bird's-Eye View Localization](https://mlsanity.com/paper/2606.07708/) (2026)
- [Generating 3D Bio-Printable Patches Using Wound Segmentation and Reconstruction to Treat Diabetic Foot Ulcers](https://mlsanity.com/paper/2203.03814/) (2022)
- [SSS3D: Fast Neural Architecture Search For Efficient Three-Dimensional Semantic Segmentation](https://mlsanity.com/paper/2304.11207/) (2023)
- [When ChatGPT for Computer Vision Will Come? From 2D to 3D](https://mlsanity.com/paper/2305.06133/) (2023)
- [Contextualising Implicit Representations for Semantic Tasks](https://mlsanity.com/paper/2305.13312/) (2023)
- [Geometry Aware Field-to-field Transformations for 3D Semantic Segmentation](https://mlsanity.com/paper/2310.05133/) (2023)
- [MSFormer: A Skeleton-multiview Fusion Method For Tooth Instance Segmentation](https://mlsanity.com/paper/2310.14489/) (2023)
- [ARCH2S: Dataset, Benchmark and Challenges for Learning Exterior Architectural Structures from Point Clouds](https://mlsanity.com/paper/2406.01337/) (2024)
- [MME: Mixture of Mesh Experts with Random Walk Transformer Gating](https://mlsanity.com/paper/2603.00828/) (2026)
- [RGB-Pointmap Pretraining for Unified 3D Scene Understanding](https://mlsanity.com/paper/2604.02546/) (2026)

## Abstract (arXiv)

3D segmentation is a fundamental and challenging problem in computer vision with applications in autonomous driving and robotics. It has received significant attention from the computer vision, graphics and machine learning communities. Conventional methods for 3D segmentation, based on hand-crafted features and machine learning classifiers, lack generalization ability. Driven by their success in 2D computer vision, deep learning techniques have recently become the tool of choice for 3D segmentation tasks. This has led to an influx of many methods in the literature that have been evaluated on different benchmark datasets. Whereas survey papers on RGB-D and point cloud segmentation exist, there is a lack of a recent in-depth survey that covers all 3D data modalities and application domains. This paper fills the gap and comprehensively surveys the recent progress in deep learning-based 3D segmentation techniques. We cover over 220 works from the last six years, analyze their strengths and limitations, and discuss their competitive results on benchmark datasets. The survey provides a summary of the most commonly used pipelines and finally highlights promising research directions for the future.