# Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery

arXiv: https://arxiv.org/abs/1903.00268 · HTML: https://mlsanity.com/paper/1903.00268/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2019.

## Built on by (most cited first, 30 of 80)

- [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024)
- [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021)
- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding](https://mlsanity.com/paper/2401.01970/) (2024)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [Kimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems](https://mlsanity.com/paper/2106.14386/) (2021)
- [Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networks](https://mlsanity.com/paper/2108.01176/) (2021)
- [3D-Aware Object Goal Navigation via Simultaneous Exploration and Identification](https://mlsanity.com/paper/2212.00338/) (2022)
- [SemGauss-SLAM: Dense Semantic Gaussian Splatting SLAM](https://mlsanity.com/paper/2403.07494/) (2024)
- [Incremental 3D Semantic Scene Graph Prediction from RGB Sequences](https://mlsanity.com/paper/2305.02743/) (2023)
- [What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models](https://mlsanity.com/paper/2512.03422/) (2025)
- [3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs](https://mlsanity.com/paper/2209.07896/) (2022)
- [RayFronts: Open-Set Semantic Ray Frontiers for Online Scene Understanding and Exploration](https://mlsanity.com/paper/2504.06994/) (2025)
- [Situational Graphs for Robot Navigation in Structured Indoor Environments](https://mlsanity.com/paper/2202.12197/) (2022)
- [NIS-SLAM: Neural Implicit Semantic RGB-D SLAM for 3D Consistent Scene Understanding](https://mlsanity.com/paper/2407.20853/) (2024)
- [360BEV: Panoramic Semantic Mapping for Indoor Bird's-Eye View](https://mlsanity.com/paper/2303.11910/) (2023)
- [Panoptic Vision-Language Feature Fields](https://mlsanity.com/paper/2309.05448/) (2023)
- [Regularized Deep Signed Distance Fields for Reactive Motion Generation](https://mlsanity.com/paper/2203.04739/) (2022)
- [Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments](https://mlsanity.com/paper/2103.16095/) (2021)
- [From SLAM to Situational Awareness: Challenges and Survey](https://mlsanity.com/paper/2110.00273/) (2021)
- [An Object SLAM Framework for Association, Mapping, and High-Level Tasks](https://mlsanity.com/paper/2305.07299/) (2023)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models](https://mlsanity.com/paper/2409.15451/) (2024)
- [OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging](https://mlsanity.com/paper/2503.01309/) (2025)
- [MEM: Multi-Modal Elevation Mapping for Robotics and Learning](https://mlsanity.com/paper/2309.16818/) (2023)
- [Leveraging Large (Visual) Language Models for Robot 3D Scene Understanding](https://mlsanity.com/paper/2209.05629/) (2022)
- [Volumetric Semantically Consistent 3D Panoptic Mapping](https://mlsanity.com/paper/2309.14737/) (2023)
- [NBV-SC: Next Best View Planning based on Shape Completion for Fruit Mapping and Reconstruction](https://mlsanity.com/paper/2209.15376/) (2022)
- [TextSLAM: Visual SLAM with Semantic Planar Text Features](https://mlsanity.com/paper/2305.10029/) (2023)
- [ObVi-SLAM: Long-Term Object-Visual SLAM](https://mlsanity.com/paper/2309.15268/) (2023)

## Abstract (arXiv)

To autonomously navigate and plan interactions in real-world environments, robots require the ability to robustly perceive and map complex, unstructured surrounding scenes. Besides building an internal representation of the observed scene geometry, the key insight toward a truly functional understanding of the environment is the usage of higher-level entities during mapping, such as individual object instances. We propose an approach to incrementally build volumetric object-centric maps during online scanning with a localized RGB-D camera. First, a per-frame segmentation scheme combines an unsupervised geometric approach with instance-aware semantic object predictions. This allows us to detect and segment elements both from the set of known classes and from other, previously unseen categories. Next, a data association step tracks the predicted instances across the different frames. Finally, a map integration strategy fuses information about their 3D shape, location, and, if available, semantic class into a global volume. Evaluation on a publicly available dataset shows that the proposed approach for building instance-level semantic maps is competitive with state-of-the-art methods, while additionally able to discover objects of unseen categories. The system is further evaluated within a real-world robotic mapping setup, for which qualitative results highlight the online nature of the method.