# Point2Graph: An End-to-end Point Cloud-based 3D Open-Vocabulary Scene Graph for Robot Navigation

arXiv: https://arxiv.org/abs/2409.10350 · HTML: https://mlsanity.com/paper/2409.10350/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 52 references that are in the map.
- First posted on arXiv 16 Sep 2024.

## Builds on (most cited first, 30 of 33)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [Uni3D: Exploring Unified 3D Representation at Scale](https://mlsanity.com/paper/2310.06773/) (2023)
- [TidyBot: Personalized Robot Assistance with Large Language Models](https://mlsanity.com/paper/2305.05658/) (2023)
- [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024)
- [Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization](https://mlsanity.com/paper/2201.13360/) (2022)
- [PLA: Language-Driven Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2211.16312/) (2022)
- [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021)
- [SAM3D: Segment Anything in 3D Scenes](https://mlsanity.com/paper/2306.03908/) (2023)
- [3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans](https://mlsanity.com/paper/2002.06289/) (2020)
- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [ShapeLLM: Universal 3D Object Understanding for Embodied Interaction](https://mlsanity.com/paper/2402.17766/) (2024)
- [Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance](https://mlsanity.com/paper/2312.10671/) (2023)
- [Grounding Language with Visual Affordances over Unstructured Data](https://mlsanity.com/paper/2210.01911/) (2022)
- [RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding](https://mlsanity.com/paper/2304.00962/) (2023)
- [OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation](https://mlsanity.com/paper/2309.00616/) (2023)
- [Foundations of Spatial Perception for Robotics: Hierarchical Representations and Real-time Systems](https://mlsanity.com/paper/2305.07154/) (2023)
- [Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries](https://mlsanity.com/paper/2211.15658/) (2022)
- [Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding](https://mlsanity.com/paper/2308.00353/) (2023)
- [V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection](https://mlsanity.com/paper/2308.04409/) (2023)

## Built on by (most cited first, 4 of 4)

- [IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigation](https://mlsanity.com/paper/2511.17384/) (2025)
- [3D Scene Graphs: Open Challenges and Future Directions](https://mlsanity.com/paper/2606.19383/) (2026)
- [M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments](https://mlsanity.com/paper/2508.17044/) (2025)
- [OpenTie: Open-vocabulary Sequential Rebar Tying System](https://mlsanity.com/paper/2509.00064/) (2025)

## Abstract (arXiv)

Current open-vocabulary scene graph generation algorithms highly rely on both 3D scene point cloud data and posed RGB-D images and thus have limited applications in scenarios where RGB-D images or camera poses are not readily available. To solve this problem, we propose Point2Graph, a novel end-to-end point cloud-based 3D open-vocabulary scene graph generation framework in which the requirement of posed RGB-D image series is eliminated. This hierarchical framework contains room and object detection/segmentation and open-vocabulary classification. For the room layer, we leverage the advantage of merging the geometry-based border detection algorithm with the learning-based region detection to segment rooms and create a "Snap-Lookup" framework for open-vocabulary room classification. In addition, we create an end-to-end pipeline for the object layer to detect and classify 3D objects based solely on 3D point cloud data. Our evaluation results show that our framework can outperform the current state-of-the-art (SOTA) open-vocabulary object and room segmentation and classification algorithm on widely used real-scene datasets.