# Towards Localizing Structural Elements: Merging Geometrical Detection with Semantic Verification in RGB-D Data

arXiv: https://arxiv.org/abs/2409.06625 · HTML: https://mlsanity.com/paper/2409.06625/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 10 Sep 2024.

## Builds on (most cited first, 11 of 11)

- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Point-SLAM: Dense Neural Point Cloud-based SLAM](https://mlsanity.com/paper/2304.04278/) (2023)
- [You Only Segment Once: Towards Real-Time Panoptic Segmentation](https://mlsanity.com/paper/2303.14651/) (2023)
- [Fully Convolutional Networks for Panoptic Segmentation with Point-based Supervision](https://mlsanity.com/paper/2108.07682/) (2021)
- [Is This The Right Place? Geometric-Semantic Pose Verification for Indoor Visual Localization](https://mlsanity.com/paper/1908.04598/) (2019)
- [Visual SLAM: What are the Current Trends and What to Expect?](https://mlsanity.com/paper/2210.10491/) (2022)
- [GeoSegNet: Point Cloud Semantic Segmentation via Geometric Encoder-Decoder Modeling](https://mlsanity.com/paper/2207.06766/) (2022)
- [Vision-based Situational Graphs Exploiting Fiducial Markers for the Integration of Semantic Entities](https://mlsanity.com/paper/2309.10461/) (2023)
- [Marker-based Visual SLAM leveraging Hierarchical Representations](https://mlsanity.com/paper/2303.01155/) (2023)
- [3D Object Detection Combining Semantic and Geometric Features from Point Clouds](https://mlsanity.com/paper/2110.04704/) (2021)

## Abstract (arXiv)

RGB-D cameras supply rich and dense visual and spatial information for various robotics tasks such as scene understanding, map reconstruction, and localization. Integrating depth and visual information can aid robots in localization and element mapping, advancing applications like 3D scene graph generation and Visual Simultaneous Localization and Mapping (VSLAM). While point cloud data containing such information is primarily used for enhanced scene understanding, exploiting their potential to capture and represent rich semantic information has yet to be adequately targeted. This paper presents a real-time pipeline for localizing building components, including wall and ground surfaces, by integrating geometric calculations for pure 3D plane detection followed by validating their semantic category using point cloud data from RGB-D cameras. It has a parallel multi-thread architecture to precisely estimate poses and equations of all the planes detected in the environment, filters the ones forming the map structure using a panoptic segmentation validation, and keeps only the validated building components. Incorporating the proposed method into a VSLAM framework confirmed that constraining the map with the detected environment-driven semantic elements can improve scene understanding and map reconstruction accuracy. It can also ensure (re-)association of these detected components into a unified 3D scene graph, bridging the gap between geometric accuracy and semantic understanding. Additionally, the pipeline allows for the detection of potential higher-level structural entities, such as rooms, by identifying the relationships between building components based on their layout.