# OneFormer3D: One Transformer for Unified Point Cloud Segmentation

arXiv: https://arxiv.org/abs/2311.14405 · HTML: https://mlsanity.com/paper/2311.14405/

## Key facts

- Cited by 95 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 52 references that are in the map.
- First posted on arXiv 24 Nov 2023.

## Builds on (most cited first, 30 of 44)

- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategies](https://mlsanity.com/paper/2206.04670/) (2022)
- [Point Transformer V2: Grouped Vector Attention and Partition-based Pooling](https://mlsanity.com/paper/2210.05666/) (2022)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling](https://mlsanity.com/paper/2003.00492/) (2020)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds](https://mlsanity.com/paper/2103.14635/) (2021)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [OccuSeg: Occupancy-aware 3D Instance Segmentation](https://mlsanity.com/paper/2003.06537/) (2020)
- [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021)
- [GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud](https://mlsanity.com/paper/1812.03320/) (2018)

## Built on by (most cited first, 30 of 95)

- [Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers](https://mlsanity.com/paper/2312.08168/) (2023)
- [Sonata: Self-Supervised Learning of Reliable Point Representations](https://mlsanity.com/paper/2503.16429/) (2025)
- [3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer](https://mlsanity.com/paper/2501.01163/) (2025)
- [EmbodiedSAM: Online Segment Any 3D Thing in Real Time](https://mlsanity.com/paper/2408.11811/) (2024)
- [SegPoint: Segment Any Point Cloud via Large Language Model](https://mlsanity.com/paper/2407.13761/) (2024)
- [DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentation](https://mlsanity.com/paper/2503.18944/) (2025)
- [Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model](https://mlsanity.com/paper/2405.17427/) (2024)
- [3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding](https://mlsanity.com/paper/2412.18450/) (2024)
- [Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2410.22489/) (2024)
- [OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splatting](https://mlsanity.com/paper/2506.07697/) (2025)
- [Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Model](https://mlsanity.com/paper/2503.16282/) (2025)
- [SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment](https://mlsanity.com/paper/2507.02705/) (2025)
- [A Unified Framework for 3D Scene Understanding](https://mlsanity.com/paper/2407.03263/) (2024)
- [ForestFormer3D: A Unified Framework for End-to-End Segmentation of Forest LiDAR 3D Point Clouds](https://mlsanity.com/paper/2506.16991/) (2025)
- [SGIFormer: Semantic-guided and Geometric-enhanced Interleaving Transformer for 3D Instance Segmentation](https://mlsanity.com/paper/2407.11564/) (2024)
- [Grounding 3D Scene Affordance From Egocentric Interactions](https://mlsanity.com/paper/2409.19650/) (2024)
- [Lost & Found: Tracking Changes from Egocentric Observations in 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2411.19162/) (2024)
- [LogoSP: Local-global Grouping of Superpoints for Unsupervised Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2506.07857/) (2025)
- [Details Matter for Indoor Open-vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2507.23134/) (2025)
- [FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videos](https://mlsanity.com/paper/2604.05621/) (2026)
- [Registration is a Powerful Rotation-Invariance Learner for 3D Anomaly Detection](https://mlsanity.com/paper/2510.16865/) (2025)
- [Swin3D++: Effective Multi-Source Pretraining for 3D Indoor Scene Understanding](https://mlsanity.com/paper/2402.14215/) (2024)
- [SAS: Segment Any 3D Scene with Integrated 2D Priors](https://mlsanity.com/paper/2503.08512/) (2025)
- [UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splatting](https://mlsanity.com/paper/2506.09952/) (2025)
- [Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale](https://mlsanity.com/paper/2506.12009/) (2025)
- [MoE3D: Mixture of Experts meets Multi-Modal 3D Understanding](https://mlsanity.com/paper/2511.22103/) (2025)
- [MICAS: Multi-grained In-Context Adaptive Sampling for 3D Point Cloud Processing](https://mlsanity.com/paper/2411.16773/) (2024)
- [CompetitorFormer: Competitor Transformer for 3D Instance Segmentation](https://mlsanity.com/paper/2411.14179/) (2024)
- [CADSpotting: Robust Panoptic Symbol Spotting on Large-Scale CAD Drawings](https://mlsanity.com/paper/2412.07377/) (2024)
- [Sketchy Bounding-box Supervision for 3D Instance Segmentation](https://mlsanity.com/paper/2505.16399/) (2025)

## Abstract (arXiv)

Semantic, instance, and panoptic segmentation of 3D point clouds have been addressed using task-specific models of distinct design. Thereby, the similarity of all segmentation tasks and the implicit relationship between them have not been utilized effectively. This paper presents a unified, simple, and effective model addressing all these tasks jointly. The model, named OneFormer3D, performs instance and semantic segmentation consistently, using a group of learnable kernels, where each kernel is responsible for generating a mask for either an instance or a semantic category. These kernels are trained with a transformer-based decoder with unified instance and semantic queries passed as an input. Such a design enables training a model end-to-end in a single run, so that it achieves top performance on all three segmentation tasks simultaneously. Specifically, our OneFormer3D ranks 1st and sets a new state-of-the-art (+2.1 mAP50) in the ScanNet test leaderboard. We also demonstrate the state-of-the-art results in semantic, instance, and panoptic segmentation of ScanNet (+21 PQ), ScanNet200 (+3.8 mAP50), and S3DIS (+0.8 mIoU) datasets.