# Pixels-to-Graph: Real-time Integration of Building Information Models and Scene Graphs for Semantic-Geometric Human-Robot Understanding

arXiv: https://arxiv.org/abs/2506.22593 · HTML: https://mlsanity.com/paper/2506.22593/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024)
- [Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization](https://mlsanity.com/paper/2201.13360/) (2022)
- [Semantic Image Inpainting with Deep Generative Models](https://mlsanity.com/paper/1607.07539/) (2016)
- [3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans](https://mlsanity.com/paper/2002.06289/) (2020)
- [3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents](https://mlsanity.com/paper/1908.04929/) (2019)
- [CubiCasa5K: A Dataset and an Improved Multi-Task Model for Floorplan Image Analysis](https://mlsanity.com/paper/1904.01920/) (2019)
- [A Unified Point-Based Framework for 3D Segmentation](https://mlsanity.com/paper/1908.00478/) (2019)
- [Bridging 2D and 3D Segmentation Networks for Computation Efficient Volumetric Medical Image Segmentation: An Empirical Study of 2.5D Solutions](https://mlsanity.com/paper/2010.06163/) (2020)
- [Autonomous Spot: Long-Range Autonomous Exploration of Extreme Environments with Legged Locomotion](https://mlsanity.com/paper/2010.09259/) (2020)
- [LOCUS 2.0: Robust and Computationally Efficient Lidar Odometry for Real-Time Underground 3D Mapping](https://mlsanity.com/paper/2205.11784/) (2022)
- [Robust Structure Identification and Room Segmentation of Cluttered Indoor Environments from Occupancy Grid Maps](https://mlsanity.com/paper/2203.03519/) (2022)
- [Learning Indoor Layouts from Simple Point-Clouds](https://mlsanity.com/paper/2108.03378/) (2021)

## Built on by (most cited first, 1 of 1)

- [MCP4IFC: IFC-Based Building Design Using Large Language Models](https://mlsanity.com/paper/2511.05533/) (2025)

## Abstract (arXiv)

Autonomous robots are increasingly playing key roles as support platforms for human operators in high-risk, dangerous applications. To accomplish challenging tasks, an efficient human-robot cooperation and understanding is required. While typically robotic planning leverages 3D geometric information, human operators are accustomed to a high-level compact representation of the environment, like top-down 2D maps representing the Building Information Model (BIM). 3D scene graphs have emerged as a powerful tool to bridge the gap between human readable 2D BIM and the robot 3D maps. In this work, we introduce Pixels-to-Graph (Pix2G), a novel lightweight method to generate structured scene graphs from image pixels and LiDAR maps in real-time for the autonomous exploration of unknown environments on resource-constrained robot platforms. To satisfy onboard compute constraints, the framework is designed to perform all operation on CPU only. The method output are a de-noised 2D top-down environment map and a structure-segmented 3D pointcloud which are seamlessly connected using a multi-layer graph abstracting information from object-level up to the building-level. The proposed method is quantitatively and qualitatively evaluated during real-world experiments performed using the NASA JPL NeBula-Spot legged robot to autonomously explore and map cluttered garage and urban office like environments in real-time.