mlsanityData to 7 Oct 2026

SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation

arXiv 2412.01801 ↗First posted 2 Dec 2024cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

We present SceneFactor, a diffusion-based approach for large-scale 3D scene generation that enables controllable generation and effortless editing. SceneFactor enables text-guided 3D scene synthesis through our factored diffusion formulation, leveraging latent semantic and geometric manifolds for generation of arbitrary-sized 3D scenes. While text input enables easy, controllable generation, text guidance remains imprecise for intuitive, localized editing and manipulation of the generated 3D scenes. Our factored semantic diffusion generates a proxy semantic space composed of semantic 3D boxes that enables controllable editing of generated scenes by adding, removing, changing the size of the semantic 3D proxy boxes that guides high-fidelity, consistent 3D geometric editing. Extensive experiments demonstrate that our approach enables high-fidelity 3D scene synthesis with effective controllable editing through our factored diffusion approach.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 76 of its 88 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

76 of 88 references.

References · 76

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  12. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
38 more
  1. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  2. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  3. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  4. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  5. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  6. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  7. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  8. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  9. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  10. 2015Empirical Evaluation of Rectified Activations in Convolutional Networkcited by 515
  11. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  12. 20203D-FRONT: 3D Furnished Rooms with layOuts and semaNTicscited by 375
  13. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  14. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  15. 20203D-FUTURE: 3D Furniture shape with TextUREcited by 306
  16. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  17. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  18. 20223D Neural Field Generation using Triplane Diffusioncited by 274
  19. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  20. 2020PolyGen: An Autoregressive Generative Model of 3D Meshescited by 267
  21. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  22. 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 254
  23. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  24. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  25. 2022Dual Diffusion Implicit Bridges for Image-to-Image Translationcited by 212
  26. 2022AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generationcited by 208
  27. 2021ATISS: Autoregressive Transformers for Indoor Scene Synthesiscited by 195
  28. 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 183
  29. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  30. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 159
  31. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  32. 2022Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functionscited by 133
  33. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generationcited by 129
  34. 2022Diffusion-SDF: Text-to-Shape via Voxelized Diffusioncited by 128
  35. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fieldscited by 122
  36. 2021NeRF-VAE: A Geometry Aware 3D Scene Generative Modelcited by 111
  37. 2022ShapeFormer: Transformer-based Shape Completion via Sparse Representationcited by 106
  38. 2021Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphscited by 83

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 10.

24 papers.

Sort:

By area:

  1. 20253D Scene Generation: A Surveycited by 24
  2. 2025NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenescited by 19
  3. 2025HiScene: Creating Hierarchical 3D Scenes with Isometric View Generationcited by 18
  4. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  5. 2025SPATIALGEN: Layout-guided 3D Indoor Scene Generationcited by 14
  6. 2026SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learningcited by 13
  7. 2025Controllable 3D Outdoor Scene Generation via Scene Graphscited by 9
  8. 2026WorldClaw: Agentic 3D Open-World Generation at Scalecited by 6
  9. 2025Towards Generating Realistic 3D Semantic Training Data for Autonomous Drivingcited by 5
  10. 2025FreeInsert: Disentangled Text-Guided Object Insertion in 3D Gaussian Scene without Spatial Priorscited by 5
  11. 2025ScanEdit: Hierarchically-Guided Functional 3D Scan Editingcited by 4
  12. 2025CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generationcited by 3
12 more
  1. 2025Action-guided generation of 3D functionality segmentation datacited by 2
  2. 2026WorldAgents: Can Foundation Image Models be Agents for 3D World Models?cited by 2
  3. 2026ShellMaker: Language-Guided Exterior Completion under Structural Constraintscited by 2
  4. 2025PrITTI: Primitive-based Generation of Controllable and Editable 3D Semantic Urban Scenescited by 1
  5. 2026Co-generation of Layout and Shape from Text via Autoregressive 3D Diffusioncited by 1
  6. 2026EditVerse3D: High-Quality 3D Object Editing with Region-Aware Learningcited by 1
  7. 2024ExCellGen: Fast, Controllable, Photorealistic 3D Scene Generation from a Single Real-World Exemplarcited by 0
  8. 2025Attacks on Approximate Caches in Text-to-Image Diffusion Modelscited by 0
  9. 2026Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flowcited by 0
  10. 2026VoxScene: Anchor-Conditioned Voxel Diffusion for Indoor Scene Arrangementcited by 0
  11. 2026HomeWorld: A Unified Floorplan-to-Furnished Framework for Generating Controllable, Densely Interactive Whole-Home Scenescited by 0
  12. 2026WorldWeave: Growing Persistent Geometric Worlds for Video Generationcited by 0