# Volumetric Semantically Consistent 3D Panoptic Mapping

arXiv: https://arxiv.org/abs/2309.14737 · HTML: https://mlsanity.com/paper/2309.14737/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 21 of 21)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [OccuSeg: Occupancy-aware 3D Instance Segmentation](https://mlsanity.com/paper/2003.06537/) (2020)
- [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021)
- [Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/1906.01140/) (2019)
- [3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2003.13867/) (2020)
- [Hierarchical Aggregation for 3D Instance Segmentation](https://mlsanity.com/paper/2108.02350/) (2021)
- [Superpoint Transformer for 3D Scene Instance Segmentation](https://mlsanity.com/paper/2211.15766/) (2022)
- [PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things](https://mlsanity.com/paper/1903.01177/) (2019)
- [Associatively Segmenting Instances and Semantics in Point Clouds](https://mlsanity.com/paper/1902.09852/) (2019)
- [Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery](https://mlsanity.com/paper/1903.00268/) (2019)
- [3D Instance Segmentation via Multi-Task Metric Learning](https://mlsanity.com/paper/1906.08650/) (2019)
- [3D-BEVIS: Bird's-Eye-View Instance Segmentation](https://mlsanity.com/paper/1904.02199/) (2019)
- [Incremental Object Database: Building 3D Models from Multiple Partial Observations](https://mlsanity.com/paper/1808.00760/) (2018)
- [Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments](https://mlsanity.com/paper/2103.16095/) (2021)

## Built on by (most cited first, 10 of 10)

- [SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs](https://mlsanity.com/paper/2404.00469/) (2024)
- [ViSTA-SLAM: Visual SLAM with Symmetric Two-view Association](https://mlsanity.com/paper/2509.01584/) (2025)
- [CoARF: Controllable 3D Artistic Style Transfer for Radiance Fields](https://mlsanity.com/paper/2404.14967/) (2024)
- [Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Description](https://mlsanity.com/paper/2412.01398/) (2024)
- [Reducing Unimodal Bias in Multi-Modal Semantic Segmentation with Multi-Scale Functional Entropy Regularization](https://mlsanity.com/paper/2505.06635/) (2025)
- [OVI-MAP:Open-Vocabulary Instance-Semantic Mapping](https://mlsanity.com/paper/2603.26541/) (2026)
- [Stable and Safe Human-aligned Reinforcement Learning through Neural Ordinary Differential Equations](https://mlsanity.com/paper/2401.13148/) (2024)
- [Open-Set 3D Semantic Instance Maps for Vision Language Navigation -- O3D-SIM](https://mlsanity.com/paper/2404.17922/) (2024)
- [RAZER: Robust Accelerated Zero-Shot 3D Open-Vocabulary Panoptic Reconstruction with Spatio-Temporal Aggregation](https://mlsanity.com/paper/2505.15373/) (2025)
- [Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations](https://mlsanity.com/paper/2506.04789/) (2025)

## Abstract (arXiv)

We introduce an online 2D-to-3D semantic instance mapping algorithm aimed at generating comprehensive, accurate, and efficient semantic 3D maps suitable for autonomous agents in unstructured environments. The proposed approach is based on a Voxel-TSDF representation used in recent algorithms. It introduces novel ways of integrating semantic prediction confidence during mapping, producing semantic and instance-consistent 3D regions. Further improvements are achieved by graph optimization-based semantic labeling and instance refinement. The proposed method achieves accuracy superior to the state of the art on public large-scale datasets, improving on a number of widely used metrics. We also highlight a downfall in the evaluation of recent studies: using the ground truth trajectory as input instead of a SLAM-estimated one substantially affects the accuracy, creating a large gap between the reported results and the actual performance on real-world data.