# SceneWiz3D: Towards Text-guided 3D Scene Composition

arXiv: https://arxiv.org/abs/2312.08885 · HTML: https://mlsanity.com/paper/2312.08885/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 62 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 30 of 50)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)

## Built on by (most cited first, 30 of 54)

- [WonderWorld: Interactive 3D Scene Generation from a Single Image](https://mlsanity.com/paper/2406.09394/) (2024)
- [CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation](https://mlsanity.com/paper/2406.02509/) (2024)
- [Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis](https://mlsanity.com/paper/2405.14868/) (2024)
- [GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting](https://mlsanity.com/paper/2402.07207/) (2024)
- [TC4D: Trajectory-Conditioned Text-to-4D Generation](https://mlsanity.com/paper/2403.17920/) (2024)
- [DreamScene: 3D Gaussian-based Text-to-3D Scene Generation via Formation Pattern Sampling](https://mlsanity.com/paper/2404.03575/) (2024)
- [SINGAPO: Single Image Controlled Generation of Articulated Parts in Objects](https://mlsanity.com/paper/2410.16499/) (2024)
- [LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation](https://mlsanity.com/paper/2408.13252/) (2024)
- [SceneCraft: Layout-Guided 3D Scene Generation](https://mlsanity.com/paper/2410.09049/) (2024)
- [Disentangled 3D Scene Generation with Layout Learning](https://mlsanity.com/paper/2402.16936/) (2024)
- [A Comprehensive Survey on 3D Content Generation](https://mlsanity.com/paper/2402.01166/) (2024)
- [Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning](https://mlsanity.com/paper/2506.05341/) (2025)
- [VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control](https://mlsanity.com/paper/2601.05138/) (2026)
- [Frankenstein: Generating Semantic-Compositional 3D Scenes in One Tri-Plane](https://mlsanity.com/paper/2403.16210/) (2024)
- [3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting](https://mlsanity.com/paper/2405.18424/) (2024)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via Stein Identity](https://mlsanity.com/paper/2401.00604/) (2023)
- [VividDream: Generating 3D Scene with Ambient Dynamics](https://mlsanity.com/paper/2405.20334/) (2024)
- [Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint](https://mlsanity.com/paper/2410.15391/) (2024)
- [GarmentDreamer: 3DGS Guided Garment Synthesis with Diverse Geometry and Texture Details](https://mlsanity.com/paper/2405.12420/) (2024)
- [PhyCAGE: Physically Plausible Compositional 3D Asset Generation from a Single Image](https://mlsanity.com/paper/2411.18548/) (2024)
- [InterGSEdit: Interactive 3D Gaussian Splatting Editing with 3D Geometry-Consistent Attention Prior](https://mlsanity.com/paper/2507.04961/) (2025)
- [HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation](https://mlsanity.com/paper/2504.13072/) (2025)
- [SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning](https://mlsanity.com/paper/2605.09423/) (2026)
- [Urban Scene Diffusion through Semantic Occupancy Map](https://mlsanity.com/paper/2403.11697/) (2024)
- [RoomTex: Texturing Compositional Indoor Scenes via Iterative Inpainting](https://mlsanity.com/paper/2406.02461/) (2024)
- [Text-to-3D Shape Generation](https://mlsanity.com/paper/2403.13289/) (2024)
- [CompGS: Unleashing 2D Compositionality for Compositional Text-to-3D via Dynamically Optimizing 3D Gaussians](https://mlsanity.com/paper/2410.20723/) (2024)
- [DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priors](https://mlsanity.com/paper/2409.08278/) (2024)
- [RoomCraft: Controllable and Complete 3D Indoor Scene Generation](https://mlsanity.com/paper/2506.22291/) (2025)

## Abstract (arXiv)

We are witnessing significant breakthroughs in the technology for generating 3D objects from text. Existing approaches either leverage large text-to-image models to optimize a 3D representation or train 3D generators on object-centric datasets. Generating entire scenes, however, remains very challenging as a scene contains multiple 3D objects, diverse and scattered. In this work, we introduce SceneWiz3D, a novel approach to synthesize high-fidelity 3D scenes from text. We marry the locality of objects with globality of scenes by introducing a hybrid 3D representation: explicit for objects and implicit for scenes. Remarkably, an object, being represented explicitly, can be either generated from text using conventional text-to-3D approaches, or provided by users. To configure the layout of the scene and automatically place objects, we apply the Particle Swarm Optimization technique during the optimization process. Furthermore, it is difficult for certain parts of the scene (e.g., corners, occlusion) to receive multi-view supervision, leading to inferior geometry. We incorporate an RGBD panorama diffusion model to mitigate it, resulting in high-quality geometry. Extensive evaluation supports that our approach achieves superior quality over previous approaches, enabling the generation of detailed and view-consistent 3D scenes.