# 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation

arXiv: https://arxiv.org/abs/2403.09439 · HTML: https://mlsanity.com/paper/2403.09439/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 68 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Builds on (most cited first, 30 of 56)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2303.08320/) (2023)
- [CLIP-Mesh: Generating textured meshes from text using pretrained image-text models](https://mlsanity.com/paper/2203.13333/) (2022)
- [Generative Novel View Synthesis with 3D-Aware Diffusion Models](https://mlsanity.com/paper/2304.02602/) (2023)

## Built on by (most cited first, 29 of 29)

- [SCube: Instant Large-Scale Scene Reconstruction using VoxSplats](https://mlsanity.com/paper/2410.20030/) (2024)
- [MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Model](https://mlsanity.com/paper/2411.16157/) (2024)
- [WonderTurbo: Generating Interactive 3D World in 0.72 Seconds](https://mlsanity.com/paper/2504.02261/) (2025)
- [FlashWorld: High-quality 3D Scene Generation within Seconds](https://mlsanity.com/paper/2510.13678/) (2025)
- [MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing](https://mlsanity.com/paper/2408.08000/) (2024)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [WorldGen: From Text to Traversable and Interactive 3D Worlds](https://mlsanity.com/paper/2511.16825/) (2025)
- [SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observations](https://mlsanity.com/paper/2505.11992/) (2025)
- [Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusion](https://mlsanity.com/paper/2507.04403/) (2025)
- [OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization](https://mlsanity.com/paper/2506.07570/) (2025)
- [Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors](https://mlsanity.com/paper/2501.02519/) (2025)
- [LAGA: Layered 3D Avatar Generation and Customization via Gaussian Splatting](https://mlsanity.com/paper/2405.12663/) (2024)
- [Scene123: One Prompt to 3D Scene Generation via Video-Assisted and Consistency-Enhanced MAE](https://mlsanity.com/paper/2408.05477/) (2024)
- [LSD-3D: Large-Scale 3D Driving Scene Generation with Geometry Grounding](https://mlsanity.com/paper/2508.19204/) (2025)
- [LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generation](https://mlsanity.com/paper/2509.05263/) (2025)
- [Constructing a 3D Scene from a Single Image](https://mlsanity.com/paper/2505.15765/) (2025)
- [PanoLora: Bridging Perspective and Panoramic Video Generation with LoRA Adaptation](https://mlsanity.com/paper/2509.11092/) (2025)
- [Self-Evolving 3D Scene Generation from a Single Image](https://mlsanity.com/paper/2512.08905/) (2025)
- [Sat2City v2: Native 3D City Asset Generation from a Single Satellite Image](https://mlsanity.com/paper/2606.24138/) (2026)
- [PaintScene4D: Consistent 4D Scene Generation from Text Prompts](https://mlsanity.com/paper/2412.04471/) (2024)
- [RoamScene3D: Immersive Text-to-3D Scene Generation via Adaptive Object-aware Roaming](https://mlsanity.com/paper/2601.19433/) (2026)
- [Sceniris: A Fast Procedural Scene Generation Framework](https://mlsanity.com/paper/2512.16896/) (2025)
- [studentSplat: Your Student Model Learns Single-view 3D Gaussian Splatting](https://mlsanity.com/paper/2601.11772/) (2026)
- [DecoRec: Decomposed 3D Scene Reconstruction from Single-View Images via Object-Level Diffusion](https://mlsanity.com/paper/2605.16807/) (2026)
- [A Cookbook of 3D Vision: Data, Learning Paradigms, and Application](https://mlsanity.com/paper/2606.04291/) (2026)
- [MagicMakeup: A Region-Controllable Diffusion Transformer for High-Fidelity Makeup-Transfer](https://mlsanity.com/paper/2607.20924/) (2026)
- [FlexComposer: Unified Video Compositing from Images to Dynamic Footage with Flexible Trajectory Control](https://mlsanity.com/paper/2607.29627/) (2026)
- [To See a World in a Living Context: Unified Indoor-Outdoor Urban World Generation](https://mlsanity.com/paper/2608.05879/) (2026)
- [Audible World Models: Spatially Aware Sound Generation for 3D Worlds](https://mlsanity.com/paper/2609.38444/) (2026)

## Abstract (arXiv)

Text-driven 3D scene generation techniques have made rapid progress in recent years. Their success is mainly attributed to using existing generative models to iteratively perform image warping and inpainting to generate 3D scenes. However, these methods heavily rely on the outputs of existing models, leading to error accumulation in geometry and appearance that prevent the models from being used in various scenarios (e.g., outdoor and unreal scenarios). To address this limitation, we generatively refine the newly generated local views by querying and aggregating global 3D information, and then progressively generate the 3D scene. Specifically, we employ a tri-plane features-based NeRF as a unified representation of the 3D scene to constrain global 3D consistency, and propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior from 2D diffusion model as well as the global 3D information of the current scene. Our extensive experiments demonstrate that, in comparison to previous methods, our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.