# Incremental 3D Semantic Scene Graph Prediction from RGB Sequences

arXiv: https://arxiv.org/abs/2305.02743 · HTML: https://mlsanity.com/paper/2305.02743/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 79 references that are in the map.
- First posted on arXiv 4 May 2023.

## Builds on (most cited first, 30 of 59)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Towards Open World Object Detection](https://mlsanity.com/paper/2103.02603/) (2021)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [3D Bounding Box Estimation Using Deep Learning and Geometry](https://mlsanity.com/paper/1612.00496/) (2016)
- [Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions](https://mlsanity.com/paper/2004.03967/) (2020)
- [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019)
- [Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization](https://mlsanity.com/paper/2201.13360/) (2022)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021)
- [3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans](https://mlsanity.com/paper/2002.06289/) (2020)
- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [Kinematic 3D Object Detection in Monocular Video](https://mlsanity.com/paper/2007.09548/) (2020)
- [CubeSLAM: Monocular 3D Object SLAM](https://mlsanity.com/paper/1806.00557/) (2018)
- [Scene Graph Generation with External Knowledge and Image Reconstruction](https://mlsanity.com/paper/1904.00560/) (2019)
- [SuMa++: Efficient LiDAR-based Semantic SLAM](https://mlsanity.com/paper/2105.11320/) (2021)

## Built on by (most cited first, 30 of 39)

- [Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships](https://mlsanity.com/paper/2402.12259/) (2024)
- [Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces](https://mlsanity.com/paper/2503.19199/) (2025)
- [Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graph](https://mlsanity.com/paper/2406.07113/) (2024)
- [OpenGraph: Open-Vocabulary Hierarchical 3D Graph Representation in Large-Scale Outdoor Environments](https://mlsanity.com/paper/2403.09412/) (2024)
- [SGRec3D: Self-Supervised 3D Scene Graph Learning via Object-Level Scene Reconstruction](https://mlsanity.com/paper/2309.15702/) (2023)
- [Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph prediction](https://mlsanity.com/paper/2310.16494/) (2023)
- [RelationField: Relate Anything in Radiance Fields](https://mlsanity.com/paper/2412.13652/) (2024)
- [SimRecon: SimReady Compositional Scene Reconstruction from Real Videos](https://mlsanity.com/paper/2603.02133/) (2026)
- [SG-PGM: Partial Graph Matching Network with Semantic Geometric Fusion for 3D Scene Graph Alignment and Its Downstream Tasks](https://mlsanity.com/paper/2403.19474/) (2024)
- [FROSS: Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Images](https://mlsanity.com/paper/2507.19993/) (2025)
- [Multiview Scene Graph](https://mlsanity.com/paper/2410.11187/) (2024)
- [Digital Twin Generation from Visual Data: A Survey](https://mlsanity.com/paper/2504.13159/) (2025)
- [OpenSU3D: Open World 3D Scene Understanding using Foundation Models](https://mlsanity.com/paper/2407.14279/) (2024)
- [ReLaGS: Relational Language Gaussian Splatting](https://mlsanity.com/paper/2603.17605/) (2026)
- [Weakly-Supervised 3D Scene Graph Generation via Visual-Linguistic Assisted Pseudo-labeling](https://mlsanity.com/paper/2404.02527/) (2024)
- [Statistical Confidence Rescoring for Robust 3D Scene Graph Generation from Multi-View Images](https://mlsanity.com/paper/2508.06546/) (2025)
- [RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.10168/) (2026)
- [FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoning](https://mlsanity.com/paper/2604.03696/) (2026)
- [Belief Scene Graphs: Expanding Partial Scenes with Objects through Computation of Expectation](https://mlsanity.com/paper/2402.03840/) (2024)
- [Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction](https://mlsanity.com/paper/2510.04714/) (2025)
- [TESGNN: Temporal Equivariant Scene Graph Neural Networks for Efficient and Robust Multi-View 3D Scene Understanding](https://mlsanity.com/paper/2411.10509/) (2024)
- [Estimating Commonsense Scene Composition on Belief Scene Graphs](https://mlsanity.com/paper/2505.02405/) (2025)
- [SceneLinker: Compositional 3D Scene Generation via Semantic Scene Graph from RGB Sequences](https://mlsanity.com/paper/2602.02974/) (2026)
- [ESGNN: Towards Equivariant Scene Graph Neural Network for 3D Scene Understanding](https://mlsanity.com/paper/2407.00609/) (2024)
- [Integrating Prior Observations for Incremental 3D Scene Graph Prediction](https://mlsanity.com/paper/2509.11895/) (2025)
- [Think While You Map: Asynchronous Vision-Language Agents for Incremental 3D Scene Graphs](https://mlsanity.com/paper/2606.31471/) (2026)
- [M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments](https://mlsanity.com/paper/2508.17044/) (2025)
- [MultiCam: On-the-fly Multi-Camera Pose Estimation Using Spatiotemporal Overlaps of Known Objects](https://mlsanity.com/paper/2603.22839/) (2026)
- [Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spaces](https://mlsanity.com/paper/2605.15753/) (2026)
- [OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environments](https://mlsanity.com/paper/2606.29786/) (2026)

## Abstract (arXiv)

3D semantic scene graphs are a powerful holistic representation as they describe the individual objects and depict the relation between them. They are compact high-level graphs that enable many tasks requiring scene reasoning. In real-world settings, existing 3D estimation methods produce robust predictions that mostly rely on dense inputs. In this work, we propose a real-time framework that incrementally builds a consistent 3D semantic scene graph of a scene given an RGB image sequence. Our method consists of a novel incremental entity estimation pipeline and a scene graph prediction network. The proposed pipeline simultaneously reconstructs a sparse point map and fuses entity estimation from the input images. The proposed network estimates 3D semantic scene graphs with iterative message passing using multi-view and geometric features extracted from the scene entities. Extensive experiments on the 3RScan dataset show the effectiveness of the proposed method in this challenging task, outperforming state-of-the-art approaches.