# SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences

arXiv: https://arxiv.org/abs/2103.14898 · HTML: https://mlsanity.com/paper/2103.14898/

## Key facts

- Cited by 163 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 64 references that are in the map.
- First posted on arXiv 27 Mar 2021.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans](https://mlsanity.com/paper/1812.07003/) (2018)
- [Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions](https://mlsanity.com/paper/2004.03967/) (2020)
- [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019)
- [Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping](https://mlsanity.com/paper/1910.02490/) (2019)
- [OccuSeg: Occupancy-aware 3D Instance Segmentation](https://mlsanity.com/paper/2003.06537/) (2020)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction](https://mlsanity.com/paper/1704.03489/) (2017)
- [3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans](https://mlsanity.com/paper/2002.06289/) (2020)
- [3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2003.13867/) (2020)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [Graph-Structured Representations for Visual Question Answering](https://mlsanity.com/paper/1609.05600/) (2016)
- [Scene Graph Generation with External Knowledge and Image Reconstruction](https://mlsanity.com/paper/1904.00560/) (2019)

## Built on by (most cited first, 30 of 163)

- [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [An Embodied Generalist Agent in 3D World](https://mlsanity.com/paper/2311.12871/) (2023)
- [3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment](https://mlsanity.com/paper/2308.04352/) (2023)
- [Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation](https://mlsanity.com/paper/2403.17846/) (2024)
- [LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark](https://mlsanity.com/paper/2306.06687/) (2023)
- [Open-vocabulary Queryable Scene Representations for Real World Planning](https://mlsanity.com/paper/2209.09874/) (2022)
- [OneFormer3D: One Transformer for Unified Point Cloud Segmentation](https://mlsanity.com/paper/2311.14405/) (2023)
- [Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs](https://mlsanity.com/paper/2108.08841/) (2021)
- [Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships](https://mlsanity.com/paper/2402.12259/) (2024)
- [3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning](https://mlsanity.com/paper/2411.17735/) (2024)
- [CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion](https://mlsanity.com/paper/2305.16283/) (2023)
- [Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces](https://mlsanity.com/paper/2503.19199/) (2025)
- [SayNav: Grounding Large Language Models for Dynamic Planning to Navigation in New Environments](https://mlsanity.com/paper/2309.04077/) (2023)
- [Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation](https://mlsanity.com/paper/2507.04047/) (2025)
- [MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenes](https://mlsanity.com/paper/2203.05203/) (2022)
- [VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud](https://mlsanity.com/paper/2303.14408/) (2023)
- [Incremental 3D Semantic Scene Graph Prediction from RGB Sequences](https://mlsanity.com/paper/2305.02743/) (2023)
- [Latent Graph Representations for Critical View of Safety Assessment](https://mlsanity.com/paper/2212.04155/) (2022)
- [GraphEQA: Using 3D Semantic Scene Graphs for Real-time Embodied Question Answering](https://mlsanity.com/paper/2412.14480/) (2024)
- [DELTA: Decomposed Efficient Long-Term Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2404.03275/) (2024)
- [(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering](https://mlsanity.com/paper/2202.09277/) (2022)
- [SGAligner : 3D Scene Alignment with Scene Graphs](https://mlsanity.com/paper/2304.14880/) (2023)
- [SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs](https://mlsanity.com/paper/2404.00469/) (2024)
- [What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Models](https://mlsanity.com/paper/2512.03422/) (2025)
- [S-Graphs+: Real-time Localization and Mapping leveraging Hierarchical Representations](https://mlsanity.com/paper/2212.11770/) (2022)
- [SPARTUN3D: Situated Spatial Understanding of 3D World in Large Language Models](https://mlsanity.com/paper/2410.03878/) (2024)
- [Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologies](https://mlsanity.com/paper/2312.11713/) (2023)
- [Bending Graphs: Hierarchical Shape Matching using Gated Optimal Transport](https://mlsanity.com/paper/2202.01537/) (2022)
- [3D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphs](https://mlsanity.com/paper/2209.07896/) (2022)

## Abstract (arXiv)

Scene graphs are a compact and explicit representation successfully used in a variety of 2D scene understanding tasks. This work proposes a method to incrementally build up semantic scene graphs from a 3D environment given a sequence of RGB-D frames. To this end, we aggregate PointNet features from primitive scene components by means of a graph neural network. We also propose a novel attention mechanism well suited for partial and missing graph data present in such an incremental reconstruction scenario. Although our proposed method is designed to run on submaps of the scene, we show it also transfers to entire 3D scenes. Experiments show that our approach outperforms 3D scene graph prediction methods by a large margin and its accuracy is on par with other 3D semantic and panoptic segmentation methods while running at 35 Hz.