mlsanityData to 14 Oct 2026

Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpainting

arXiv 2404.19758 ↗First posted 30 Apr 2024cs.CVcited by 24

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

3D scene generation has quickly become a challenging new research direction, fueled by consistent improvements of 2D generative diffusion models. Most prior work in this area generates scenes by iteratively stitching newly generated frames with existing geometry. These works often depend on pre-trained monocular depth estimators to lift the generated images into 3D, fusing them with the existing scene representation. These approaches are then often evaluated via a text metric, measuring the similarity between the generated images and a given text prompt. In this work, we make two fundamental contributions to the field of 3D scene generation. First, we note that lifting images to 3D with a monocular depth estimation model is suboptimal as it ignores the geometry of the existing scene. We thus introduce a novel depth completion model, trained via teacher distillation and self-training to learn the 3D fusion process, resulting in improved geometric coherence of the scene. Second, we introduce a new benchmarking scheme for scene generation methods that is based on ground truth geometry, and thus measures the quality of the structure of the scene.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 66 of its 75 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

66 of 75 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation4 Dec
  2. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  3. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors30 Jun
  4. 2023The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation2 Jun
  5. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  6. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields19 May
  7. 2023LDM3D: Latent Diffusion Model for 3D18 May
  8. 20233D-aware Image Generation using 2D Diffusion Models31 Mar
  9. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  10. 2022RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models12 Dec
  11. 2022DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Models22 Nov
  12. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
1 more
  1. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  8. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  9. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  10. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
  11. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  12. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
38 more
  1. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 735
  2. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  3. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  4. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 657
  5. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
  6. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  7. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  8. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 405
  9. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  10. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 364
  11. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
  12. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  13. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 243
  14. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  15. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 231
  16. 2020Learning to Recover 3D Scene Shape from a Single Imagecited by 216
  17. 20203D Photography using Context-aware Layered Depth Inpaintingcited by 206
  18. 2020Non-Local Spatial Propagation Network for Depth Completioncited by 195
  19. 2020GRF: Learning a General Radiance Field for 3D Representation and Renderingcited by 183
  20. 2021Deep Parametric Continuous Convolutional Neural Networkscited by 161
  21. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  22. 2023WonderJourney: Going from Anywhere to Everywherecited by 158
  23. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  24. 2021MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesiscited by 133
  25. 2018Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Networkcited by 110
  26. 2019CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completioncited by 110
  27. 2023CompletionFormer: Depth Completion with Convolutions and Vision Transformerscited by 93
  28. 2024BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolationcited by 83
  29. 2018Layer-structured 3D Scene Inference via View Synthesiscited by 82
  30. 2021PixelSynth: Generating a 3D-Consistent Experience from a Single Imagecited by 82
  31. 2022InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Imagescited by 82
  32. 2020Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Imagecited by 77
  33. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  34. 2020Learning Joint 2D-3D Representations for Depth Completioncited by 71
  35. 2019HoloGAN: Unsupervised learning of 3D representations from natural imagescited by 59
  36. 2023Designing a Better Asymmetric VQGAN for StableDiffusioncited by 45
  37. 2023PERF: Panoramic Neural Radiance Field from a Single Panoramacited by 45
  38. 2023Text2Immersion: Generative Immersive Scene with 3D Gaussianscited by 43

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 14; 2026: 5.

24 papers.

Sort:

By area:

  1. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 202
  2. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 156
  3. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  4. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 40
  5. 2025WonderTurbo: Generating Interactive 3D World in 0.72 Secondscited by 32
  6. 2025SynCity: Training-Free Generation of 3D Worldscited by 31
  7. 20253D Scene Generation: A Surveycited by 24
  8. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 19
  9. 2026Latent Spatial Memory for Video World Modelscited by 17
  10. 2025SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observationscited by 16
  11. 2025TeleWorld: Towards Dynamic Multimodal Synthesis with a 4D World Modelcited by 16
  12. 2024HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Modelcited by 9
12 more
  1. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 8
  2. 2024Amodal Depth Anything: Amodal Depth Estimation in the Wildcited by 5
  3. 2025CoCo4D: Comprehensive and Complex 4D Scene Generationcited by 4
  4. 2025BloomScene: Lightweight Structured 3D Gaussian Splatting for Crossmodal Scene Generationcited by 3
  5. 2025Constructing a 3D Scene from a Single Imagecited by 3
  6. 2025Self-Evolving 3D Scene Generation from a Single Imagecited by 3
  7. 2024PanoDreamer: Optimization-Based Single Image to 360 3D Scene With Diffusioncited by 2
  8. 2025Dream-to-Recon: Monocular 3D Reconstruction with Diffusion-Depth Distillation from Single Imagescited by 1
  9. 2026OctWorld: Long-Range World-Consistent Video Generation with Octree-Based 3D Mappingcited by 1
  10. 2025Towards Fingerprint Mosaicking Artifact Detection: A Self-Supervised Deep Learning Approachcited by 0
  11. 2026Interpretable Human-Label-Free Deep Learning for Real-Bogus Classification with Uncertainty Quantificationcited by 0
  12. 20264LC: 4D Latent Cache for Efficient Conditioning in Dynamic World Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion
  2. 2025SceneCompleter: Dense 3D Scene Completion for Generative Novel View Synthesis
  3. 2025Geometry-Aware Diffusion Models for Multiview Scene Inpainting
  4. 20243D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation
  5. 2023NeRFiller: Completing Scenes via Generative 3D Inpainting
  6. 2025G4Splat: Geometry-Guided Gaussian Splatting with Generative Prior
  7. 2025GaussVideoDreamer: 3D Scene Generation with Video Diffusion and Inconsistency-Aware Gaussian Splatting
  8. 2025Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors