mlsanityData to 7 Oct 2026

Frankenstein: Generating Semantic-Compositional 3D Scenes in One Tri-Plane

arXiv 2403.16210 ↗First posted 24 Mar 2024cs.CV cs.AI cs.GRcited by 26

Not on the map. Why?

Abstract (arXiv)

We present Frankenstein, a diffusion-based framework that can generate semantic-compositional 3D scenes in a single pass. Unlike existing methods that output a single, unified 3D shape, Frankenstein simultaneously generates multiple separated shapes, each corresponding to a semantically meaningful part. The 3D scene information is encoded in one single tri-plane tensor, from which multiple Singed Distance Function (SDF) fields can be decoded to represent the compositional shapes. During training, an auto-encoder compresses tri-planes into a latent space, and then the denoising diffusion process is employed to approximate the distribution of the compositional scenes. Frankenstein demonstrates promising results in generating room interiors as well as human avatars with automatically separated parts. The generated scenes facilitate many downstream applications, such as part-wise re-texturing, object rearrangement in the room or avatar cloth re-targeting. Our project page is available at: https://wolfball.github.io/frankenstein/.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 47 of its 66 references that are in the map. First posted on arXiv 24 Mar 2024.

Builds on

47 of 66 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  6. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  7. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  8. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Modelcited by 516
  9. 2023Neuralangelo: High-Fidelity Neural Surface Reconstructioncited by 464
  10. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  11. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 411
  12. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
35 more
  1. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  2. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusioncited by 309
  3. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 306
  4. 20223D Neural Field Generation using Triplane Diffusioncited by 274
  5. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  6. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  7. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  8. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  9. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generationcited by 181
  10. 2022Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functionscited by 133
  11. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generationcited by 129
  12. 2022Diffusion-SDF: Text-to-Shape via Voxelized Diffusioncited by 128
  13. 2023TADA! Text to Animatable Digital Avatarscited by 109
  14. 2018Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Modelscited by 105
  15. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  16. 2023GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphscited by 96
  17. 2020House-GAN: Relational Generative Adversarial Networks for Graph-constrained House Layout Generationcited by 86
  18. 2023Text-Guided Texturing by Synchronized Multi-View Diffusioncited by 86
  19. 2023AnyHome: Open-Vocabulary Generation of Structured and Textured 3D Homescited by 81
  20. 2024BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolationcited by 81
  21. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  22. 2022Object-Compositional Neural Implicit Surfacescited by 74
  23. 2023Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraintscited by 59
  24. 2023CC3D: Layout-Conditioned Generation of Compositional 3D Scenescited by 58
  25. 2019SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentationcited by 55
  26. 2023ControlRoom3D: Room Generation using Semantic Proxy Roomscited by 54
  27. 2023SceneWiz3D: Towards Text-guided 3D Scene Compositioncited by 54
  28. 2023Learning Disentangled Avatars with Hybrid 3D Representationscited by 44
  29. 2023ObjectSDF++: Improved Object-Compositional Neural Implicit Surfacescited by 38
  30. 2023Disentangled Clothed Avatar Generation from Text Descriptionscited by 33
  31. 2024Disentangled 3D Scene Generation with Layout Learningcited by 33
  32. 2023HumanLiff: Layer-wise 3D Human Generation with Diffusion Modelcited by 23
  33. 2023Advances in Data-Driven Analysis and Synthesis of 3D Indoor Scenescited by 19
  34. 2023PAniC-3D: Stylized Single-view 3D Reconstruction from Portraits of Anime Characterscited by 16
  35. 2023AssetField: Assets Mining and Reconfiguration in Ground Feature Plane Representationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 9; 2026: 9.

26 papers.

Sort:

By area:

  1. 2024REPARO: Compositional 3D Assets Generation with Differentiable 3D Layout Alignmentcited by 45
  2. 2025X-Part: high fidelity and structure coherent shape decompositioncited by 42
  3. 2024SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generationcited by 24
  4. 20253D Scene Generation: A Surveycited by 24
  5. 2025DepR: Depth Guided Single-view Scene Reconstruction with Instance-level Diffusioncited by 23
  6. 2024Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketchescited by 21
  7. 2024PhyCAGE: Physically Plausible Compositional 3D Asset Generation from a Single Imagecited by 19
  8. 2025MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generationcited by 18
  9. 2025HiScene: Creating Hierarchical 3D Scenes with Isometric View Generationcited by 18
  10. 2024CompGS: Unleashing 2D Compositionality for Compositional Text-to-3D via Dynamically Optimizing 3D Gaussianscited by 10
  11. 2024Semantic Score Distillation Sampling for Compositional Text-to-3D Generationcited by 9
  12. 2025Efficient Multi-Camera Tokenization with Triplanes for End-to-End Drivingcited by 6
14 more
  1. 2025Part-X-MLLM: Part-aware 3D Multimodal Large Language Modelcited by 6
  2. 2026WorldClaw: Agentic 3D Open-World Generation at Scalecited by 6
  3. 2024StdGEN: Semantic-Decomposed 3D Character Generation from Single Imagescited by 5
  4. 2025PartDiffuser: Part-wise 3D Mesh Generation via Discrete Diffusioncited by 4
  5. 2026FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flowcited by 4
  6. 2025CHARM: Control-point-based 3D Anime Hairstyle Auto-Regressive Modelingcited by 3
  7. 2026Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modelingcited by 2
  8. 2026StdGEN++: A Comprehensive System for Semantic-Decomposed 3D Character Generationcited by 1
  9. 2026DM-CFO: A Diffusion Model for Compositional 3D Tooth Generation with Collision-Free Optimizationcited by 1
  10. 2026Co-generation of Layout and Shape from Text via Autoregressive 3D Diffusioncited by 1
  11. 2024Fused-Planes: Why Train a Thousand Tri-Planes When You Can Share?cited by 0
  12. 2026Muses: Designing, Composing, Generating Nonexistent Fantasy 3D Creatures without Trainingcited by 0
  13. 2026Hunyuan3D-Buffalo 1.0: A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editingcited by 0
  14. 2026SAM3D-Part: Interactive Part Selection and Generation from 3D Objectscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers
  2. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Models
  3. 2024LT3SD: Latent Trees for 3D Scene Diffusion
  4. 2025Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors
  5. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis
  6. 2024Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy
  7. 2026Structured 3D Latents Are Surprisingly Powerful: Unleashing Generalizable Style with 2D Diffusion
  8. 2025From One to More: Contextual Part Latents for 3D Generation