mlsanityData to 14 Oct 2026

External Knowledge Enhanced 3D Scene Generation from Sketch

arXiv 2403.14121 ↗First posted 21 Mar 2024cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Generating realistic 3D scenes is challenging due to the complexity of room layouts and object geometries.We propose a sketch based knowledge enhanced diffusion architecture (SEK) for generating customized, diverse, and plausible 3D scenes. SEK conditions the denoising process with a hand-drawn sketch of the target scene and cues from an object relationship knowledge base. We first construct an external knowledge base containing object relationships and then leverage knowledge enhanced graph reasoning to assist our model in understanding hand-drawn sketches. A scene is represented as a combination of 3D objects and their relationships, and then incrementally diffused to reach a Gaussian distribution.We propose a 3D denoising scene transformer that learns to reverse the diffusion process, conditioned by a hand-drawn sketch along with knowledge cues, to regressively generate the scene including the 3D object instances as well as their layout. Experiments on the 3D-FRONT dataset show that our model improves FID, CKL by 17.41%, 37.18% in 3D scene generation and FID, KID by 19.12%, 20.06% in 3D scene completion compared to the nearest competitor DiffuScene.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 42 of its 58 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

42 of 58 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  6. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  7. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2020Center-based 3D Object Detection and Trackingcited by 1,144
  10. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  11. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 380
  12. 2018Image Generation from Scene Graphscited by 324
30 more
  1. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  2. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 208
  3. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 200
  4. 2022Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimizationcited by 178
  5. 2020Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Imagecited by 172
  6. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 158
  7. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  8. 2016Graph-Structured Representations for Visual Question Answeringcited by 123
  9. 2019LayoutVAE: Stochastic Scene Layout Generation From a Label Setcited by 119
  10. 2022Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentationcited by 119
  11. 2018Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Modelscited by 106
  12. 2019Scene Graph Generation with External Knowledge and Image Reconstructioncited by 102
  13. 2020Bridging Knowledge Graphs to Generate Scene Graphscited by 86
  14. 2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 84
  15. 2023SKED: Sketch-guided Text-based 3D Editingcited by 73
  16. 2022Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Trainingcited by 56
  17. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  18. 2021Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketchescited by 54
  19. 2020End-to-End Optimization of Scene Layoutcited by 52
  20. 2017Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scenecited by 50
  21. 2023Spectrum-guided Multi-granularity Referring Video Object Segmentationcited by 48
  22. 20173D Shape Reconstruction from Sketches via Multi-view Convolutional Networkscited by 45
  23. 2023SceneHGN: Hierarchical Graph Networks for 3D Indoor Scene Generation with Fine-Grained Geometrycited by 39
  24. 2022MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answeringcited by 36
  25. 2023Sketch and Text Guided Diffusion Model for Colored Point Cloud Generationcited by 36
  26. 2021Scene Synthesis via Uncertainty-Driven Attribute Synchronizationcited by 29
  27. 2021Indoor Scene Generation from a Collection of Semantic-Segmented Depth Imagescited by 27
  28. 2019DeepSketchHair: Deep Sketch-based 3D Hair Modelingcited by 25
  29. 2022Learning 3D Scene Priors with 2D Supervisioncited by 24
  30. 2023Sketch-A-Shape: Zero-Shot Sketch-to-3D Shape Generationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 3.

9 papers.

  1. 2025HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generationcited by 24
  2. 20253D Scene Generation: A Surveycited by 24
  3. 2024Articulate3D: Holistic Understanding of 3D Scenes as Universal Scene Descriptioncited by 9
  4. 2024S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Fieldcited by 3
  5. 2025CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generationcited by 3
  6. 2026LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigationcited by 2
  7. 2024Navigate Complex Physical Worlds via Geometrically Constrained LLMcited by 0
  8. 2026Class-Partitioned VQ-VAE and Latent Flow Matching for Point Cloud Scene Generationcited by 0
  9. 2026GridFlow: Structured Latent Flow for Seamless City-Scale 3D Point Cloud Generationcited by 0