mlsanityData to 7 Oct 2026

Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs

arXiv 2108.08841 ↗First posted 19 Aug 2021cs.CVcited by 83

Not on the map. Why?

Abstract (arXiv)

Controllable scene synthesis consists of generating 3D information that satisfy underlying specifications. Thereby, these specifications should be abstract, i.e. allowing easy user interaction, whilst providing enough interface for detailed control. Scene graphs are representations of a scene, composed of objects (nodes) and inter-object relationships (edges), proven to be particularly suited for this task, as they allow for semantic control on the generated content. Previous works tackling this task often rely on synthetic data, and retrieve object meshes, which naturally limits the generation capabilities. To circumvent this issue, we instead propose the first work that directly generates shapes from a scene graph in an end-to-end manner. In addition, we show that the same model supports scene modification, using the respective scene graph as interface. Leveraging Graph Convolutional Networks (GCN) we train a variational Auto-Encoder on top of the object and edge categories, as well as 3D shapes and scene layouts, allowing latter sampling of new scenes and shapes.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 19 Aug 2021.

Builds on

33 of 43 references.

References · 33

  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  6. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  7. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 562
  8. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  9. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  10. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  11. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  12. 2018Image Generation from Scene Graphscited by 324
21 more
  1. 2018Graph R-CNN for Scene Graph Generationcited by 296
  2. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  3. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  4. 2019RIO: 3D Object Instance Re-Localization in Changing Indoor Environmentscited by 187
  5. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 170
  6. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 159
  7. 2017Scene Graph Generation from Objects, Phrases and Region Captionscited by 154
  8. 2019LayoutVAE: Stochastic Scene Layout Generation From a Label Setcited by 119
  9. 2020Semantic Image Manipulation Using Scene Graphscited by 82
  10. 2018Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generationcited by 78
  11. 2019Specifying Object Attributes and Relations in Interactive Scene Generationcited by 76
  12. 2018Attentive Relational Networks for Mapping Images to Scene Graphscited by 72
  13. 2019StructureNet: Hierarchical Graph Networks for 3D Shape Generationcited by 59
  14. 2017Pixels to Graphs by Associative Embeddingcited by 58
  15. 2019SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentationcited by 55
  16. 2020End-to-End Optimization of Scene Layoutcited by 52
  17. 2017Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scenecited by 50
  18. 2018Mapping Images to Scene Graphs with Permutation-Invariant Structured Predictioncited by 44
  19. 2018GRAINS: Generative Recursive Autoencoders for INdoor Scenescited by 38
  20. 2017Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammarscited by 25
  21. 2019Hierarchy Denoising Recursive Autoencoders for 3D Scene Layout Predictioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 15; 2024: 17; 2025: 25; 2026: 17.

83 papers.

Sort:

By area:

  1. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 116
  2. 2024InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Priorcited by 110
  3. 2023GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphscited by 96
  4. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understandingcited by 95
  5. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  6. 2022DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Imagescited by 69
  7. 2024EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusioncited by 66
  8. 2022A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspectivecited by 56
  9. 2023CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusioncited by 54
  10. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  11. 2023Incremental 3D Semantic Scene Graph Prediction from RGB Sequencescited by 39
  12. 2022ScanEnts3D: Exploiting Phrase-to-3D-Object Correspondences for Improved Visio-Linguistic Models in 3D Scenescited by 32
38 more
  1. 2025Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlookcited by 32
  2. 2024DiffAssemble: A Unified Graph-Diffusion Model for 2D and 3D Reassemblycited by 31
  3. 2023SGAligner : 3D Scene Alignment with Scene Graphscited by 30
  4. 2025Global-Local Tree Search in VLMs for 3D Indoor Scene Generationcited by 29
  5. 2025What Is The Best 3D Scene Representation for Robotics? From Geometric to Foundation Modelscited by 29
  6. 2022S-Graphs+: Real-time Localization and Mapping leveraging Hierarchical Representationscited by 28
  7. 2023SGFormer: Semantic Graph Transformer for Point Cloud-based 3D Scene Graph Generationcited by 27
  8. 2023NAP: Neural 3D Articulation Priorcited by 25
  9. 2023Indoor and Outdoor 3D Scene Graph Generation via Language-Enabled Spatial Ontologiescited by 25
  10. 20223D VSG: Long-term Semantic Scene Change Prediction through 3D Variable Scene Graphscited by 24
  11. 2023SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene Graphscited by 24
  12. 2024SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generationcited by 24
  13. 20253D Scene Generation: A Surveycited by 24
  14. 2025HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generationcited by 22
  15. 2023SGRec3D: Self-Supervised 3D Scene Graph Learning via Object-Level Scene Reconstructioncited by 20
  16. 2024Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLMcited by 20
  17. 2023Graph Transformer GANs for Graph-Constrained House Generationcited by 19
  18. 2023Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph predictioncited by 18
  19. 2025MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generationcited by 18
  20. 2024"Where am I?" Scene Retrieval with Languagecited by 15
  21. 2025A Causal Adjustment Module for Debiasing Scene Graph Generationcited by 14
  22. 2023Language-driven Scene Synthesis using Multi-conditional Diffusion Modelcited by 10
  23. 2024SG-PGM: Partial Graph Matching Network with Semantic Geometric Fusion for 3D Scene Graph Alignment and Its Downstream Taskscited by 10
  24. 2025La La LiDAR: Large-Scale Layout Generation from LiDAR Datacited by 10
  25. 2022Spatial Commonsense Graph for Object Localisation in Partial Scenescited by 9
  26. 2024Planner3D: LLM-enhanced graph prior meets 3D indoor scene explicit regularizationcited by 9
  27. 2024External Knowledge Enhanced 3D Scene Generation from Sketchcited by 9
  28. 2025Controllable 3D Outdoor Scene Generation via Scene Graphscited by 9
  29. 2025FreeScene: Mixed Graph Diffusion for 3D Scene Synthesis from Free Promptscited by 8
  30. 2021MIGS: Meta Image Generation from Scene Graphscited by 7
  31. 2024Lay-A-Scene: Personalized 3D Object Arrangement Using Text-to-Image Priorscited by 7
  32. 2024InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis with Semantic Graph Priorcited by 6
  33. 2025Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimizationcited by 6
  34. 2025From Programs to Poses: Factored Real-World Scene Generation via Learned Program Librariescited by 6
  35. 2024Graph Transformer GANs with Graph Masked Modeling for Architectural Layout Generationcited by 5
  36. 2025PRISM-0: A Predicate-Rich Scene Graph Generation Framework for Zero-Shot Open-Vocabulary Taskscited by 5
  37. 2025ReSpace: Text-Driven Autoregressive 3D Indoor Scene Synthesis and Editingcited by 5
  38. 2025Statistical Confidence Rescoring for Robust 3D Scene Graph Generation from Multi-View Imagescited by 5

50 most cited of 83 All 83 (JSON)