mlsanityData to 7 Oct 2026

Disentangled 3D Scene Generation with Layout Learning

arXiv 2402.16936 ↗First posted 26 Feb 2024cs.CV cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

We introduce a method to generate 3D scenes that are disentangled into their component objects. This disentanglement is unsupervised, relying only on the knowledge of a large pretrained text-to-image model. Our key insight is that objects can be discovered by finding parts of a 3D scene that, when rearranged spatially, still produce valid configurations of the same scene. Concretely, our method jointly optimizes multiple NeRFs from scratch - each representing its own object - along with a set of layouts that composite these objects into scenes. We then encourage these composited scenes to be in-distribution according to the image generator. We show that despite its simplicity, our approach successfully generates 3D scenes decomposed into individual objects, enabling new capabilities in text-to-3D content creation. For results and an interactive demo, see our project page at https://dave.ml/layoutlearning/

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 43 of its 59 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

43 of 59 references.

References · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Segment Anythingcited by 8,158
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  9. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  10. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  11. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
31 more
  1. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  2. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  3. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  4. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  5. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  6. 2020Object-Centric Learning with Slot Attentioncited by 865
  7. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  8. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  9. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 641
  10. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  11. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  12. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  13. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  14. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 323
  15. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
  16. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  17. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  18. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 211
  19. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  20. 2023Unsupervised Semantic Correspondence Using Stable Diffusioncited by 120
  21. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101
  22. 2021Unsupervised Discovery of Object Radiance Fieldscited by 100
  23. 2016Learning Visual N-Grams from Web Datacited by 95
  24. 2020The Hessian Penalty: A Weak Prior for Unsupervised Disentanglementcited by 90
  25. 2022Object Scene Representation Transformercited by 90
  26. 2022Object discovery and representation networkscited by 80
  27. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  28. 2022Deformable Sprites for Unsupervised Video Decompositioncited by 61
  29. 2023SceneWiz3D: Towards Text-guided 3D Scene Compositioncited by 54
  30. 2023Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitivescited by 37
  31. 2022Unsupervised Discovery and Composition of Object Light Fieldscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 10; 2026: 6.

33 papers.

Sort:

By area:

  1. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  2. 2024LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Modelscited by 94
  3. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 76
  4. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Modelscited by 60
  5. 2024ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidancecited by 59
  6. 2024REPARO: Compositional 3D Assets Generation with Differentiable 3D Layout Alignmentcited by 45
  7. 2025WonderTurbo: Generating Interactive 3D World in 0.72 Secondscited by 31
  8. 2024Frankenstein: Generating Semantic-Compositional 3D Scenes in One Tri-Planecited by 26
  9. 20243DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splattingcited by 25
  10. 2024AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animationcited by 25
  11. 20253D Scene Generation: A Surveycited by 24
  12. 2024Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketchescited by 21
21 more
  1. 2025WorldCraft: Photo-Realistic 3D World Creation and Customization via LLM Agentscited by 21
  2. 2024BrightDreamer: Generic 3D Gaussian Generative Framework for Fast Text-to-3D Synthesiscited by 20
  3. 2024PhyCAGE: Physically Plausible Compositional 3D Asset Generation from a Single Imagecited by 19
  4. 2025MMGDreamer: Mixed-Modality Graph for Geometry-Controllable 3D Indoor Scene Generationcited by 18
  5. 2025Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusioncited by 15
  6. 2025Agentic 3D Scene Generation with Spatially Contextualized VLMscited by 14
  7. 2024Semantic Score Distillation Sampling for Compositional Text-to-3D Generationcited by 9
  8. 2024DreamDissector: Learning Disentangled Text-to-3D Generation from 2D Diffusion Priorscited by 8
  9. 2024Scene123: One Prompt to 3D Scene Generation via Video-Assisted and Consistency-Enhanced MAEcited by 6
  10. 2025Towards Geometric and Textural Consistency 3D Scene Generation via Single Image-guided Model Generation and Layout Optimizationcited by 6
  11. 2024DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfercited by 5
  12. 2026FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flowcited by 4
  13. 20253D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worldscited by 3
  14. 2025DreamAnywhere: Object-Centric Panoramic 3D Scene Generationcited by 2
  15. 2026Sat2City v2: Native 3D City Asset Generation from a Single Satellite Imagecited by 2
  16. 2026RoamScene3D: Immersive Text-to-3D Scene Generation via Adaptive Object-aware Roamingcited by 1
  17. 2026Seeing Through Clutter: Structured 3D Scene Reconstruction via Iterative Object Removalcited by 1
  18. 2026HOG-Layout: Hierarchical 3D Scene Generation, Optimization and Editing via Vision-Language Modelscited by 1
  19. 2024ExCellGen: Fast, Controllable, Photorealistic 3D Scene Generation from a Single Real-World Exemplarcited by 0
  20. 2025AnchorHOI: Zero-shot Generation of 4D Human-Object Interaction via Anchor-based Prior Distillationcited by 0
  21. 2026Scene Generation at Absolute Scale: Utilizing Semantic and Geometric Guidance From Text for Accurate and Interpretable 3D Indoor Scene Generationcited by 0