mlsanityData to 7 Oct 2026

RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion

arXiv 2404.07199 ↗First posted 10 Apr 2024cs.CV cs.AI cs.GRcited by 109

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We introduce RealmDreamer, a technique for generating forward-facing 3D scenes from text descriptions. Our method optimizes a 3D Gaussian Splatting representation to match complex text prompts using pretrained diffusion models. Our key insight is to leverage 2D inpainting diffusion models conditioned on an initial scene estimate to provide low variance supervision for unknown regions during 3D distillation. In conjunction, we imbue high-fidelity geometry with geometric distillation from a depth diffusion model, conditioned on samples from the inpainting model. We find that the initialization of the optimization is crucial, and provide a principled methodology for doing so. Notably, our technique doesn't require video or multi-view data and can synthesize various high-quality 3D scenes in different styles with complex layouts. Further, the generality of our method allows 3D synthesis from a single image. As measured by a comprehensive user study, our method outperforms all existing approaches, preferred by 88-95%. Project Page: https://realmdreamer.github.io/

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 71 of its 92 references that are in the map. First posted on arXiv 10 Apr 2024.

Builds on

71 of 92 references.

Closest ancestors · 20

Same direction, newest first

  1. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation4 Dec
  2. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model23 Oct
  3. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models12 Oct
  4. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  5. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  6. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors30 Jun
  7. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization29 Jun
  8. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidance30 May
  9. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  10. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields19 May
  11. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Models5 Apr
  12. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
8 more
  1. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  2. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  3. 2022RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models12 Dec
  4. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors6 Dec
  5. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  6. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  7. 2022Novel View Synthesis with Diffusion Models6 Oct
  8. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
38 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  5. 2016Improved Techniques for Training GANscited by 2,938
  6. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  7. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  8. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  9. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  10. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  11. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  12. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  13. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  14. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  15. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  16. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 596
  17. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  18. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  19. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 482
  20. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  21. 2023Text-to-3D using Gaussian Splattingcited by 299
  22. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  23. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  24. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  25. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  26. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 228
  27. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  28. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 179
  29. 2018Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddingscited by 169
  30. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  31. 2022SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fieldscited by 144
  32. 2023Text-to-Image Diffusion Models are Zero-Shot Classifierscited by 140
  33. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  34. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
  35. 2021PixelSynth: Generating a 3D-Consistent Experience from a Single Imagecited by 82
  36. 2023Reference-guided Controllable Inpainting of Neural Radiance Fieldscited by 40
  37. 2015Text to 3D Scene Generation with Rich Lexical Groundingcited by 36
  38. 2023iNVS: Repurposing Diffusion Inpainters for Novel View Synthesiscited by 20

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 59; 2026: 19.

109 papers.

Sort:

By area:

  1. 2024Depth Anything V2cited by 1,385
  2. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  3. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
  4. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  5. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  6. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 98
  7. 2024GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warpingcited by 72
  8. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  9. 2025Bolt3D: Generating 3D Scenes in Secondscited by 57
  10. 2024SCube: Instant Large-Scale Scene Reconstruction using VoxSplatscited by 56
  11. 20244K4DGen: Panoramic 4D Generation at 4K Resolutioncited by 54
  12. 2024Director3D: Real-world Camera Trajectory and 3D Scene Generation from Textcited by 50
38 more
  1. 2024VistaDream: Sampling multiview consistent images for single-view scene reconstructioncited by 38
  2. 2025FlexWorld: Progressively Expanding 3D Scenes for Flexiable-View Synthesiscited by 37
  3. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  4. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 36
  5. 2024DreamDrive: Generative 4D Scene Modeling from Street View Imagescited by 35
  6. 2024SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesiscited by 34
  7. 2024Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generationcited by 33
  8. 2024MVGenMaster: Scaling Multi-View Generation from Any Image via 3D Priors Enhanced Diffusion Modelcited by 31
  9. 2025WonderTurbo: Generating Interactive 3D World in 0.72 Secondscited by 31
  10. 2025SynCity: Training-Free Generation of 3D Worldscited by 30
  11. 2025FlashWorld: High-quality 3D Scene Generation within Secondscited by 30
  12. 2025WorldExplorer: Towards Generating Fully Navigable 3D Scenescited by 26
  13. 2024SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splattingcited by 24
  14. 20253D Scene Generation: A Surveycited by 24
  15. 2024From an Image to a Scene: Learning to Imagine the World from a Million 360 Videoscited by 23
  16. 2024Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketchescited by 21
  17. 2025Positional Encoding Fieldcited by 20
  18. 2025Generative Gaussian Splatting: Generating 3D Scenes with Video Diffusion Priorscited by 19
  19. 2025NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenescited by 19
  20. 2025SuperDec: 3D Scene Decomposition with Superquadric Primitivescited by 19
  21. 2025A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generationcited by 19
  22. 2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
  23. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  24. 2024Diffusion-Based Visual Art Creation: A Survey and New Perspectivescited by 17
  25. 2024Advancing Extended Reality with 3D Gaussian Splatting: Innovations and Prospectscited by 17
  26. 2025Orchid: Image Latent Diffusion for Joint Appearance and Geometry Generationcited by 17
  27. 2024Pano2Room: Novel View Synthesis from a Single Indoor Panoramacited by 16
  28. 2025Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidancecited by 16
  29. 2025Compositional Generative Model of Unbounded 4D Citiescited by 16
  30. 2025SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observationscited by 16
  31. 2025Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagerycited by 16
  32. 2026WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memoriescited by 15
  33. 2025PAT3D: Physics-Augmented Text-to-3D Scene Generationcited by 14
  34. 2025A Recipe for Generating 3D Worlds From a Single Imagecited by 13
  35. 2025Scene Splatter: Momentum 3D Scene Generation from Single Image with Video Diffusion Modelcited by 13
  36. 2025VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting Generation with Flexible Pose and Multi-View Joint Modelingcited by 12
  37. 2025LAYOUTDREAMER: Physics-guided Layout for Text-to-3D Compositional Scene Generationcited by 11
  38. 2025Scene4U: Hierarchical Layered 3D Scene Reconstruction from Single Panoramic Image for Your Immerse Explorationcited by 11

50 most cited of 109 All 109 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors
  2. 20243D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation
  3. 2023GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs
  4. 2023EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior
  5. 2026Co-generation of Layout and Shape from Text via Autoregressive 3D Diffusion
  6. 2026SIC3D: Style Image Conditioned Text-to-3D Gaussian Splatting Generation
  7. 2024VividDreamer: Towards High-Fidelity and Efficient Text-to-3D Generation
  8. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation