# JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds with Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields

arXiv: https://arxiv.org/abs/1904.00699 · HTML: https://mlsanity.com/paper/1904.00699/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2019.

## Built on by (most cited first, 30 of 62)

- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [Self-Supervised Pretraining of 3D Features on any Point-Cloud](https://mlsanity.com/paper/2101.02691/) (2021)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [Hierarchical Aggregation for 3D Instance Segmentation](https://mlsanity.com/paper/2108.02350/) (2021)
- [NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes](https://mlsanity.com/paper/2111.13260/) (2021)
- [Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation](https://mlsanity.com/paper/2103.14962/) (2021)
- [RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving](https://mlsanity.com/paper/2301.10222/) (2023)
- [Learning 3D Semantic Segmentation with only 2D Image Supervision](https://mlsanity.com/paper/2110.11325/) (2021)
- [Panoptic-PHNet: Towards Real-Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmap](https://mlsanity.com/paper/2205.07002/) (2022)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR-based Perception](https://mlsanity.com/paper/2109.05441/) (2021)
- [ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution](https://mlsanity.com/paper/2303.00246/) (2023)
- [Point-set Distances for Learning Representations of 3D Point Clouds](https://mlsanity.com/paper/2102.04014/) (2021)
- [Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervision](https://mlsanity.com/paper/2303.03361/) (2023)
- [Deep Learning Based 3D Segmentation: A Survey](https://mlsanity.com/paper/2103.05423/) (2021)
- [UrbanBIS: a Large-scale Benchmark for Fine-grained Urban Building Instance Segmentation](https://mlsanity.com/paper/2305.02627/) (2023)
- [DRINet++: Efficient Voxel-as-point Point Cloud Segmentation](https://mlsanity.com/paper/2111.08318/) (2021)
- [Omni-supervised Point Cloud Segmentation via Gradual Receptive Field Component Reasoning](https://mlsanity.com/paper/2105.10203/) (2021)
- [Learning with Noisy Labels for Robust Point Cloud Segmentation](https://mlsanity.com/paper/2107.14230/) (2021)
- [Point Scene Understanding via Disentangled Instance Mesh Reconstruction](https://mlsanity.com/paper/2203.16832/) (2022)
- [PartSTAD: 2D-to-3D Part Segmentation Task Adaptation](https://mlsanity.com/paper/2401.05906/) (2024)
- [PointInst3D: Segmenting 3D Instances by Points](https://mlsanity.com/paper/2204.11402/) (2022)
- [Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments](https://mlsanity.com/paper/2103.16095/) (2021)
- [Point Cloud Instance Segmentation with Semi-supervised Bounding-Box Mining](https://mlsanity.com/paper/2111.15210/) (2021)
- [Scalable SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2209.08263/) (2022)
- [3D-QueryIS: A Query-based Framework for 3D Instance Segmentation](https://mlsanity.com/paper/2211.09375/) (2022)
- [ForestFormer3D: A Unified Framework for End-to-End Segmentation of Forest LiDAR 3D Point Clouds](https://mlsanity.com/paper/2506.16991/) (2025)
- [3D Instance Segmentation of MVS Buildings](https://mlsanity.com/paper/2112.09902/) (2021)
- [LatticeNet: Fast Spatio-Temporal Point Cloud Segmentation Using Permutohedral Lattices](https://mlsanity.com/paper/2108.03917/) (2021)
- [LiDAR-based 4D Panoptic Segmentation via Dynamic Shifting Network](https://mlsanity.com/paper/2203.07186/) (2022)
- [CP3: Channel Pruning Plug-in for Point-based Networks](https://mlsanity.com/paper/2303.13097/) (2023)

## Abstract (arXiv)

Deep learning techniques have become the to-go models for most vision-related tasks on 2D images. However, their power has not been fully realised on several tasks in 3D space, e.g., 3D scene understanding. In this work, we jointly address the problems of semantic and instance segmentation of 3D point clouds. Specifically, we develop a multi-task pointwise network that simultaneously performs two tasks: predicting the semantic classes of 3D points and embedding the points into high-dimensional vectors so that points of the same object instance are represented by similar embeddings. We then propose a multi-value conditional random field model to incorporate the semantic and instance labels and formulate the problem of semantic and instance segmentation as jointly optimising labels in the field model. The proposed method is thoroughly evaluated and compared with existing methods on different indoor scene datasets including S3DIS and SceneNN. Experimental results showed the robustness of the proposed joint semantic-instance segmentation scheme over its single components. Our method also achieved state-of-the-art performance on semantic segmentation.