mlsanityData to 14 Oct 2026

Incremental 3D Semantic Scene Graph Prediction from RGB Sequences

arXiv 2305.02743 ↗First posted 4 May 2023cs.CVcited by 39

Not on the map. Why?

Abstract (arXiv)

3D semantic scene graphs are a powerful holistic representation as they describe the individual objects and depict the relation between them. They are compact high-level graphs that enable many tasks requiring scene reasoning. In real-world settings, existing 3D estimation methods produce robust predictions that mostly rely on dense inputs. In this work, we propose a real-time framework that incrementally builds a consistent 3D semantic scene graph of a scene given an RGB image sequence. Our method consists of a novel incremental entity estimation pipeline and a scene graph prediction network. The proposed pipeline simultaneously reconstructs a sparse point map and fuses entity estimation from the input images. The proposed network estimates 3D semantic scene graphs with iterative message passing using multi-view and geometric features extracted from the scene entities. Extensive experiments on the 3RScan dataset show the effectiveness of the proposed method in this challenging task, outperforming state-of-the-art approaches.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 59 of its 79 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

59 of 79 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  6. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  9. 2020ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAMcited by 878
  10. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 565
  11. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  12. 2017Scene Graph Generation by Iterative Message Passingcited by 462
38 more
  1. 2016Visual Relationship Detection with Language Priorscited by 334
  2. 2018Image Generation from Scene Graphscited by 324
  3. 2018Graph R-CNN for Scene Graph Generationcited by 298
  4. 2021Towards Open World Object Detectioncited by 297
  5. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  6. 20163D Bounding Box Estimation Using Deep Learning and Geometrycited by 278
  7. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 208
  8. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 188
  9. 2022Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimizationcited by 178
  10. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 172
  11. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 163
  12. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 144
  13. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  14. 2016SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networkscited by 135
  15. 2020Kinematic 3D Object Detection in Monocular Videocited by 116
  16. 2018CubeSLAM: Monocular 3D Object SLAMcited by 102
  17. 2019Scene Graph Generation with External Knowledge and Image Reconstructioncited by 102
  18. 2021SuMa++: Efficient LiDAR-based Semantic SLAMcited by 100
  19. 2018Fusion++: Volumetric Object-Level SLAMcited by 98
  20. 2019PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Thingscited by 98
  21. 2018MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objectscited by 95
  22. 2021Holistic 3D Scene Understanding from a Single Image with Implicit Representationcited by 94
  23. 2020Fully Convolutional Networks for Panoptic Segmentationcited by 92
  24. 2018QuadricSLAM: Dual Quadrics from Object Detections as Landmarks in Object-oriented SLAMcited by 86
  25. 2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 84
  26. 2019Unpaired Image Captioning via Scene Graph Alignmentscited by 82
  27. 2020Semantic Image Manipulation Using Scene Graphscited by 82
  28. 2019Volumetric Instance-Aware Semantic Mapping and 3D Object Discoverycited by 80
  29. 2018Joint Monocular 3D Vehicle Detection and Trackingcited by 77
  30. 20193-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agentscited by 74
  31. 2021Open-World Entity Segmentationcited by 74
  32. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  33. 2021Fully Convolutional Scene Graph Generationcited by 57
  34. 2021Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networkscited by 52
  35. 2018A Unified Framework for Multi-View Multi-Class Object Pose Estimationcited by 49
  36. 20224D-OR: Semantic Scene Graphs for OR Domain Modelingcited by 49
  37. 2020Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentationcited by 40
  38. 2018Visual Graphs from Motion (VGfM): Scene understanding with object geometry reasoningcited by 27

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 11; 2025: 8; 2026: 18.

39 papers.

Sort:

By area:

  1. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 80
  2. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 54
  3. 2024Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graphcited by 32
  4. 2024OpenGraph: Open-Vocabulary Hierarchical 3D Graph Representation in Large-Scale Outdoor Environmentscited by 22
  5. 2023SGRec3D: Self-Supervised 3D Scene Graph Learning via Object-Level Scene Reconstructioncited by 21
  6. 2023Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph predictioncited by 19
  7. 2024RelationField: Relate Anything in Radiance Fieldscited by 16
  8. 2026SimRecon: SimReady Compositional Scene Reconstruction from Real Videoscited by 12
  9. 2024SG-PGM: Partial Graph Matching Network with Semantic Geometric Fusion for 3D Scene Graph Alignment and Its Downstream Taskscited by 10
  10. 2025FROSS: Faster-than-Real-Time Online 3D Semantic Scene Graph Generation from RGB-D Imagescited by 10
  11. 2024Multiview Scene Graphcited by 9
  12. 2025Digital Twin Generation from Visual Data: A Surveycited by 7
27 more
  1. 2024OpenSU3D: Open World 3D Scene Understanding using Foundation Modelscited by 6
  2. 2026ReLaGS: Relational Language Gaussian Splattingcited by 6
  3. 2024Weakly-Supervised 3D Scene Graph Generation via Visual-Linguistic Assisted Pseudo-labelingcited by 5
  4. 2025Statistical Confidence Rescoring for Robust 3D Scene Graph Generation from Multi-View Imagescited by 5
  5. 2026RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generationcited by 5
  6. 2026FunFact: Building Probabilistic Functional 3D Scene Graphs via Factor-Graph Reasoningcited by 5
  7. 2024Belief Scene Graphs: Expanding Partial Scenes with Objects through Computation of Expectationcited by 4
  8. 2025Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Predictioncited by 4
  9. 2024TESGNN: Temporal Equivariant Scene Graph Neural Networks for Efficient and Robust Multi-View 3D Scene Understandingcited by 3
  10. 2025Estimating Commonsense Scene Composition on Belief Scene Graphscited by 3
  11. 2026SceneLinker: Compositional 3D Scene Generation via Semantic Scene Graph from RGB Sequencescited by 3
  12. 2024ESGNN: Towards Equivariant Scene Graph Neural Network for 3D Scene Understandingcited by 2
  13. 2025Integrating Prior Observations for Incremental 3D Scene Graph Predictioncited by 2
  14. 2026Think While You Map: Asynchronous Vision-Language Agents for Incremental 3D Scene Graphscited by 2
  15. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  16. 2026MultiCam: On-the-fly Multi-Camera Pose Estimation Using Spatiotemporal Overlaps of Known Objectscited by 1
  17. 2026Hierarchical and Holistic Open-Vocabulary Functional 3D Scene Graphs for Indoor Spacescited by 1
  18. 2026OP3DSG: Open-Vocabulary Part-Aware 3D Scene Graph Generation for Real-World Environmentscited by 1
  19. 2026DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priorscited by 1
  20. 2026R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Videocited by 1
  21. 2026MA3DSG: Multi-Agent 3D Scene Graph Generation for Large-Scale Indoor Environmentscited by 0
  22. 2026OpenSGA: Efficient 3D Scene Graph Alignment in the Open Worldcited by 0
  23. 2026SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matchingcited by 0
  24. 2026SGFormer++: Semantic Graph Transformer for Incremental 3D Scene Graph Generationcited by 0
  25. 2026NoPA: Non-Parametric Online 3D Scene Graph Generationcited by 0
  26. 2026PUF: Plug-and-Play Uncertainty-Aware Fusion for Online 3D Scene Graph Generationcited by 0
  27. 2026GraphWrit3R: End-to-End 3D Scene Graph Writingcited by 0