# FROSS: Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Images arXiv: https://arxiv.org/abs/2507.19993 ยท HTML: https://mlsanity.com/paper/2507.19993/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 35 of its 46 references that are in the map. - First posted on arXiv 26 Jul 2025. ## Builds on (most cited first, 30 of 35) - [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016) - [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016) - [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016) - [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019) - [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020) - [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023) - [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017) - [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017) - [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023) - [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018) - [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019) - [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018) - [Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions](https://mlsanity.com/paper/2004.03967/) (2020) - [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019) - [Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization](https://mlsanity.com/paper/2201.13360/) (2022) - [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021) - [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021) - [Relation-Aware Graph Attention Network for Visual Question Answering](https://mlsanity.com/paper/1903.12314/) (2019) - [GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs](https://mlsanity.com/paper/2312.00093/) (2023) - [RT-DETRv2: Improved Baseline with Bag-of-Freebies for Real-Time Detection Transformer](https://mlsanity.com/paper/2407.17140/) (2024) - [I-Design: Personalized LLM Interior Designer](https://mlsanity.com/paper/2404.02838/) (2024) - [ViNG: Learning Open-World Navigation with Visual Goals](https://mlsanity.com/paper/2012.09812/) (2020) - [3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents](https://mlsanity.com/paper/1908.04929/) (2019) - [EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusion](https://mlsanity.com/paper/2405.00915/) (2024) - [CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion](https://mlsanity.com/paper/2305.16283/) (2023) - [Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysis](https://mlsanity.com/paper/2103.05558/) (2021) - [EGTR: Extracting Graph from Transformer for Scene Graph Generation](https://mlsanity.com/paper/2404.02072/) (2024) - [VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud](https://mlsanity.com/paper/2303.14408/) (2023) - [Incremental 3D Semantic Scene Graph Prediction from RGB Sequences](https://mlsanity.com/paper/2305.02743/) (2023) - [Dense Relational Captioning: Triple-Stream Networks for Relationship-Based Captioning](https://mlsanity.com/paper/1903.05942/) (2019) ## Built on by (most cited first, 10 of 10) - [RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.10168/) (2026) - [3D Scene Graphs: Open Challenges and Future Directions](https://mlsanity.com/paper/2606.19383/) (2026) - [The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models](https://mlsanity.com/paper/2601.08876/) (2026) - [Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering](https://mlsanity.com/paper/2510.01483/) (2025) - [OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environments](https://mlsanity.com/paper/2606.29786/) (2026) - [DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors](https://mlsanity.com/paper/2607.00889/) (2026) - [DGA-Net: Enhancing SAM with Depth Prompting and Graph-Anchor Guidance for Camouflaged Object Detection](https://mlsanity.com/paper/2601.02831/) (2026) - [NoPA: Non-Parametric Online 3D Scene Graph Generation](https://mlsanity.com/paper/2607.00529/) (2026) - [PUF: Plug-and-Play Uncertainty-Aware Fusion for Online 3D Scene Graph Generation](https://mlsanity.com/paper/2607.07170/) (2026) - [Prior-SG: Task and Prior Driven Region Segmentation for Scene Graphs in Arbitrarily-Structured Environments](https://mlsanity.com/paper/2608.06170/) (2026) ## Abstract (arXiv) The ability to abstract complex 3D environments into simplified and structured representations is crucial across various domains. 3D semantic scene graphs (SSGs) achieve this by representing objects as nodes and their interrelationships as edges, facilitating high-level scene understanding. Existing methods for 3D SSG generation, however, face significant challenges, including high computational demands and non-incremental processing that hinder their suitability for real-time open-world applications. To address this issue, we propose FROSS (Faster-than-Real-Time Online 3D Semantic Scene Graph Generation), an innovative approach for online and faster-than-real-time 3D SSG generation that leverages the direct lifting of 2D scene graphs to 3D space and represents objects as 3D Gaussian distributions. This framework eliminates the dependency on precise and computationally-intensive point cloud processing. Furthermore, we extend the Replica dataset with inter-object relationship annotations, creating the ReplicaSSG dataset for comprehensive evaluation of FROSS. The experimental results from evaluations on ReplicaSSG and 3DSSG datasets show that FROSS can achieve superior performance while operating significantly faster than prior 3D SSG generation methods. Our implementation and dataset are publicly available at https://github.com/Howardkhh/FROSS.