mlsanityData to 7 Oct 2026

3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation

arXiv 2403.09439 ↗First posted 14 Mar 2024cs.CV cs.AIcited by 29

Not on the map. Why?

Abstract (arXiv)

Text-driven 3D scene generation techniques have made rapid progress in recent years. Their success is mainly attributed to using existing generative models to iteratively perform image warping and inpainting to generate 3D scenes. However, these methods heavily rely on the outputs of existing models, leading to error accumulation in geometry and appearance that prevent the models from being used in various scenarios (e.g., outdoor and unreal scenarios). To address this limitation, we generatively refine the newly generated local views by querying and aggregating global 3D information, and then progressively generate the 3D scene. Specifically, we employ a tri-plane features-based NeRF as a unified representation of the 3D scene to constrain global 3D consistency, and propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior from 2D diffusion model as well as the global 3D information of the current scene. Our extensive experiments demonstrate that, in comparison to previous methods, our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 56 of its 68 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

56 of 68 references.

References · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  6. 2022Video Diffusion Modelscited by 2,280
  7. 2021Vision Transformers for Dense Predictioncited by 1,789
  8. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  10. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  11. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  12. 2015Spatial Transformer Networkscited by 1,423
38 more
  1. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  2. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  3. 2022BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformerscited by 1,213
  4. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  5. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  6. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  7. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  8. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  9. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  10. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  11. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  12. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  13. 2021iMAP: Implicit Mapping and Positioning in Real-Timecited by 476
  14. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  15. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  16. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  17. 2022CLIP-Mesh: Generating textured meshes from text using pretrained image-text modelscited by 281
  18. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  19. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  20. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  21. 2018Learning Blind Video Temporal Consistencycited by 247
  22. 2023SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesiscited by 247
  23. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  24. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 239
  25. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 210
  26. 20203D Photography using Context-aware Layered Depth Inpaintingcited by 206
  27. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  28. 2017Video Generation From Textcited by 153
  29. 2023Infinite Photorealistic Worlds using Procedural Generationcited by 142
  30. 2022NeRF-Art: Text-Driven Neural Radiance Fields Stylizationcited by 126
  31. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fieldscited by 122
  32. 2021Pathdreamer: A World Model for Indoor Navigationcited by 117
  33. 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
  34. 2023Towards Zero-Shot Scale-Aware Monocular Depth Estimationcited by 100
  35. 2022NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstructioncited by 99
  36. 2018To Create What You Tell: Generating Videos from Captionscited by 84
  37. 2021PixelSynth: Generating a 3D-Consistent Experience from a Single Imagecited by 82
  38. 2022InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Imagescited by 82

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 14; 2026: 9.

29 papers.

Sort:

By area:

  1. 2024SCube: Instant Large-Scale Scene Reconstruction using VoxSplatscited by 56
  2. 2024MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Modelcited by 31
  3. 2025WonderTurbo: Generating Interactive 3D World in 0.72 Secondscited by 31
  4. 2024MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editingcited by 30
  5. 2025FlashWorld: High-quality 3D Scene Generation within Secondscited by 30
  6. 20253D Scene Generation: A Surveycited by 24
  7. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  8. 2025SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observationscited by 16
  9. 2025Sat2City: 3D City Generation from A Single Satellite Image with Cascaded Latent Diffusioncited by 15
  10. 2025OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimizationcited by 14
  11. 2025Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priorscited by 10
  12. 2024LAGA: Layered 3D Avatar Generation and Customization via Gaussian Splattingcited by 7
17 more
  1. 2024Scene123: One Prompt to 3D Scene Generation via Video-Assisted and Consistency-Enhanced MAEcited by 6
  2. 2025LSD-3D: Large-Scale 3D Driving Scene Generation with Geometry Groundingcited by 5
  3. 2025LatticeWorld: A Multimodal Large Language Model-Empowered Framework for Interactive Complex World Generationcited by 5
  4. 2025Self-Evolving 3D Scene Generation from a Single Imagecited by 3
  5. 2024PaintScene4D: Consistent 4D Scene Generation from Text Promptscited by 2
  6. 2025Constructing a 3D Scene from a Single Imagecited by 2
  7. 2025PanoLora: Bridging Perspective and Panoramic Video Generation with LoRA Adaptationcited by 2
  8. 2026Sat2City v2: Native 3D City Asset Generation from a Single Satellite Imagecited by 2
  9. 2026RoamScene3D: Immersive Text-to-3D Scene Generation via Adaptive Object-aware Roamingcited by 1
  10. 2025Sceniris: A Fast Procedural Scene Generation Frameworkcited by 0
  11. 2026studentSplat: Your Student Model Learns Single-view 3D Gaussian Splattingcited by 0
  12. 2026DecoRec: Decomposed 3D Scene Reconstruction from Single-View Images via Object-Level Diffusioncited by 0
  13. 2026A Cookbook of 3D Vision: Data, Learning Paradigms, and Applicationcited by 0
  14. 2026MagicMakeup: A Region-Controllable Diffusion Transformer for High-Fidelity Makeup-Transfercited by 0
  15. 2026FlexComposer: Unified Video Compositing from Images to Dynamic Footage with Flexible Trajectory Controlcited by 0
  16. 2026To See a World in a Living Context: Unified Indoor-Outdoor Urban World Generationcited by 0
  17. 2026Audible World Models: Spatially Aware Sound Generation for 3D Worldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TRELLISWorld: Training-Free World Generation from Object Generators
  2. 2026Co-generation of Layout and Shape from Text via Autoregressive 3D Diffusion
  3. 2024VividDreamer: Towards High-Fidelity and Efficient Text-to-3D Generation
  4. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion
  5. 2025Setting the Stage: Text-Driven Scene-Consistent Image Generation
  6. 2026SceneAssistant: A Visual Feedback Agent for Open-Vocabulary 3D Scene Generation
  7. 2026Any 3D Scene is Worth 1K Tokens: 3D-Grounded Representation for Scene Generation at Scale
  8. 2025Recent Advances in 3D Object and Scene Generation: A Survey