mlsanityData to 14 Oct 2026

Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenes

arXiv 2303.13450 ↗First posted 23 Mar 2023cs.CV cs.GR cs.LGcited by 75

Not on the map. Why?

Cited by 23 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent breakthroughs in text-guided image generation have led to remarkable progress in the field of 3D synthesis from text. By optimizing neural radiance fields (NeRF) directly from text, recent methods are able to produce remarkable results. Yet, these methods are limited in their control of each object's placement or appearance, as they represent the scene as a whole. This can be a major issue in scenarios that require refining or manipulating objects in the scene. To remedy this deficit, we propose a novel GlobalLocal training framework for synthesizing a 3D scene using object proxies. A proxy represents the object's placement in the generated scene and optionally defines its coarse geometry. The key to our approach is to represent each object as an independent NeRF. We alternate between optimizing each NeRF on its own and as part of the full scene. Thus, a complete representation of each object can be learned, while also creating a harmonious scene with style and lighting match. We show that using proxies allows a wide variety of editing options, such as adjusting the placement of each independent object, removing objects from a scene, or refining an object. Our results show that Set-the-Scene offers a powerful solution for scene synthesis and manipulation, filling a crucial gap in controllable text-to-3D synthesis.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Builds on 25 of its 29 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

25 of 29 references.

References · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  5. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  6. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
  7. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
  8. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  9. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  10. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  11. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  12. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 334
13 more
  1. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  2. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  3. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  4. 2021Learning Object-Compositional Neural Radiance Field for Editable Scene Renderingcited by 228
  5. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  6. 2022Compressible-composable NeRF via Rank-residual Decompositioncited by 92
  7. 2020Object-Centric Neural Scene Renderingcited by 90
  8. 2022TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decompositioncited by 83
  9. 2023SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collectionscited by 72
  10. 2022DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesiscited by 61
  11. 2023Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervisioncited by 44
  12. 2022Simple and Effective Synthesis of Indoor 3D Scenescited by 33
  13. 2023Text-driven Visual Synthesis with Latent Diffusion Priorcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 31; 2025: 26; 2026: 6.

75 papers.

Sort:

By area:

  1. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
  2. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 231
  3. 2024InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Priorcited by 117
  4. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 109
  5. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 77
  6. 2024DreamScene: 3D Gaussian-based Text-to-3D Scene Generation via Formation Pattern Samplingcited by 72
  7. 2024EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusioncited by 67
  8. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Modelscited by 62
  9. 2023Generative AI meets 3D: A Survey on Text-to-3D in AIGC Eracited by 61
  10. 2023ControlRoom3D: Room Generation using Semantic Proxy Roomscited by 56
  11. 2023SceneWiz3D: Towards Text-guided 3D Scene Compositioncited by 54
  12. 2024LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generationcited by 53
38 more
  1. 2024Director3D: Real-world Camera Trajectory and 3D Scene Generation from Textcited by 50
  2. 2024SceneTeller: Language-to-3D Scene Generationcited by 43
  3. 2024SceneCraft: Layout-Guided 3D Scene Generationcited by 41
  4. 2024HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splattingcited by 35
  5. 2024Disentangled 3D Scene Generation with Layout Learningcited by 33
  6. 2025SynCity: Training-Free Generation of 3D Worldscited by 31
  7. 2025ArtiScene: Language-Driven Artistic 3D Scene Generation Through Image Intermediarycited by 28
  8. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 28
  9. 2025DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimizationcited by 27
  10. 2024Frankenstein: Generating Semantic-Compositional 3D Scenes in One Tri-Planecited by 26
  11. 2025WorldExplorer: Towards Generating Fully Navigable 3D Scenescited by 26
  12. 2024Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpaintingcited by 24
  13. 2024FastScene: Text-Driven Fast 3D Indoor Scene Generation via Panoramic Gaussian Splattingcited by 24
  14. 2024SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splattingcited by 24
  15. 20253D Scene Generation: A Surveycited by 24
  16. 2024VividDream: Generating 3D Scene with Ambient Dynamicscited by 21
  17. 2023DreamPropeller: Supercharge Text-to-3D Generation with Parallel Samplingcited by 20
  18. 2024InterFusion: Text-Driven Generation of 3D Human-Object Interactioncited by 18
  19. 2023Towards Language-guided Interactive 3D Generation: LLMs as Layout Interpreter with Generative Feedbackcited by 17
  20. 2024Urban Architect: Steerable 3D Urban Scene Generation with Layout Priorcited by 17
  21. 2025Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusioncited by 16
  22. 2024RoomTex: Texturing Compositional Indoor Scenes via Iterative Inpaintingcited by 14
  23. 2025SPATIALGEN: Layout-guided 3D Indoor Scene Generationcited by 14
  24. 2024Text-to-3D Shape Generationcited by 13
  25. 2024SceneGenAgent: Precise Industrial Scene Generation with Coding Agentcited by 13
  26. 2025Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priorscited by 13
  27. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  28. 2025Neural Radiance Fields for the Real World: A Surveycited by 12
  29. 2024SIGNeRF: Scene Integrated Generation for Neural Radiance Fieldscited by 11
  30. 2023LucidDreaming: Controllable Object-Centric 3D Generationcited by 10
  31. 2024CompGS: Unleashing 2D Compositionality for Compositional Text-to-3D via Dynamically Optimizing 3D Gaussianscited by 10
  32. 2024RoomPainter: View-Integrated Diffusion for Consistent Indoor Scene Texturingcited by 10
  33. 2024Semantically-aware Neural Radiance Fields for Visual Scene Understanding: A Comprehensive Reviewcited by 9
  34. 2025Zero-1-to-A: Zero-Shot One Image to Animatable Head Avatars Using Video Diffusioncited by 9
  35. 2024DreamDissector: Learning Disentangled Text-to-3D Generation from 2D Diffusion Priorscited by 8
  36. 2024FlexGen: Flexible Multi-View Generation from Text and Image Inputscited by 8
  37. 2026SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creationcited by 8
  38. 2023FaceDNeRF: Semantics-Driven Face Reconstruction, Prompt Editing and Relighting with Diffusion Modelscited by 7

50 most cited of 75 All 75 (JSON)