mlsanityData to 14 Oct 2026

RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models

arXiv 2212.05993 ↗First posted 12 Dec 2022cs.CVcited by 45

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We address the challenge of recovering an underlying scene geometry and colors from a sparse set of RGBD view observations. In this work, we present a new solution termed RGBD2 that sequentially generates novel RGBD views along a camera trajectory, and the scene geometry is simply the fusion result of these views. More specifically, we maintain an intermediate surface mesh used for rendering new RGBD views, which subsequently becomes complete by an inpainting network; each rendered RGBD view is later back-projected as a partial surface and is supplemented into the intermediate mesh. The use of intermediate mesh and camera projection helps solve the tough problem of multi-view inconsistency. We practically implement the RGBD inpainting network as a versatile RGBD diffusion model, which is previously used for 2D generative modeling; we make a modification to its reverse diffusion process to enable our use. We evaluate our approach on the task of 3D scene synthesis from sparse RGBD inputs; extensive experiments on the ScanNet dataset demonstrate the superiority of our approach over existing ones. Project page: https://jblei.site/proj/rgbd-diffusion.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 82 of its 99 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

82 of 99 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Novel View Synthesis with Diffusion Models6 Oct
  2. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  3. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan
  4. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 78

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,847
  2. 2022Generative Adversarial Networkscited by 5,627
  3. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  4. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  8. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  9. 2022Video Diffusion Modelscited by 2,305
  10. 2015Variational Inference with Normalizing Flowscited by 2,170
  11. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  13. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,712
  14. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  15. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  16. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  17. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  18. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 927
  19. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  20. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 735
  21. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  22. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
  23. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 689
  24. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 651
  25. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 642
  26. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 615
  27. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 588
  28. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  29. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  30. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  31. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  32. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 488
  33. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  34. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  35. 2019DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstructioncited by 333
  36. 2021Neural RGB-D Surface Reconstructioncited by 313
  37. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 309
  38. 2021Dense Depth Priors for Neural Radiance Fields from Sparse Input Viewscited by 302

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 18; 2025: 12; 2026: 6.

45 papers.

Sort:

By area:

  1. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 689
  2. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 645
  3. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 231
  4. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 202
  5. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 166
  6. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusioncited by 109
  7. 20233D-aware Image Generation using 2D Diffusion Modelscited by 69
  8. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  9. 2024SemCity: Semantic Scene Generation with Triplane Diffusioncited by 53
  10. 2024Director3D: Real-world Camera Trajectory and 3D Scene Generation from Textcited by 50
  11. 2024VistaDream: Sampling multiview consistent images for single-view scene reconstructioncited by 38
  12. 2024Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generationcited by 34
33 more
  1. 2025FlashWorld: High-quality 3D Scene Generation within Secondscited by 32
  2. 2025SynCity: Training-Free Generation of 3D Worldscited by 31
  3. 2024Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpaintingcited by 24
  4. 2024SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splattingcited by 24
  5. 2025RORem: Training a Robust Object Remover with Human-in-the-Loopcited by 23
  6. 2023GCD-DDPM: A Generative Change Detection Model Based on Difference-Feature Guided DDPMcited by 21
  7. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 19
  8. 2024Guess The Unseen: Dynamic 3D Scene Reconstruction from Partial 2D Glimpsescited by 17
  9. 2024Pano2Room: Novel View Synthesis from a Single Indoor Panoramacited by 16
  10. 2024ViewFusion: Towards Multi-View Consistency via Interpolated Denoisingcited by 14
  11. 2025Dynamic View Synthesis as an Inverse Problemcited by 12
  12. 2023Learning Neural Implicit through Volume Rendering with Attentive Depth Fusion Priorscited by 11
  13. 2024GenRC: Generative 3D Room Completion from Sparse Image Collectionscited by 9
  14. 2025Wonder3D++: Cross-domain Diffusion for High-fidelity 3D Generation from a Single Imagecited by 8
  15. 2025IDCNet: Guided Video Diffusion for Metric-Consistent RGBD Scene Generation with Precise Camera Controlcited by 7
  16. 2026PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangementcited by 7
  17. 2024Direct and Explicit 3D Generation from a Single Imagecited by 6
  18. 2026AnchoredDream: Zero-Shot 360deg Indoor Scene Generation from a Single View via Geometric Groundingcited by 5
  19. 2024The Adversarial AI-Art: Understanding, Generation, Detection, and Benchmarkingcited by 4
  20. 2024PointRegGPT: Boosting 3D Point Cloud Registration using Generative Point-Cloud Pairs for Trainingcited by 4
  21. 2025Geometry and Perception Guided Gaussians for Multiview-consistent 3D Generation from a Single Imagecited by 4
  22. 2025DMS:Diffusion-Based Multi-Baseline Stereo Generation for Improving Self-Supervised Depth Estimationcited by 3
  23. 2025GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generationcited by 3
  24. 2023SparseGNV: Generating Novel Views of Indoor Scenes with Sparse Input Viewscited by 2
  25. 2024MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview and Textcited by 2
  26. 2024RoofDiffusion: Constructing Roofs from Severely Corrupted Point Data via Diffusioncited by 2
  27. 2025Voyaging into Perpetual Dynamic Scenes from a Single Viewcited by 2
  28. 2025Programmable-Room: Interactive Textured 3D Room Meshes Generation Empowered by Large Language Modelscited by 1
  29. 2026RoamScene3D: Immersive Text-to-3D Scene Generation via Adaptive Object-aware Roamingcited by 1
  30. 2026VidSplat: Gaussian Splatting Reconstruction with Geometry-Guided Video Diffusion Priorscited by 1
  31. 2023Optimized View and Geometry Distillation from Multi-view Diffusercited by 0
  32. 2026CGGS: Consistency-Augmented Geometric Gaussian Splatting for Ego-Centric 3D Scene Generationcited by 0
  33. 2026GSComplete: Gaussian Splat Completion with 2D Diffusion Priorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MultiDiff: Consistent Novel View Synthesis from a Single Image
  2. 2024RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models
  3. 2024Coherent 3D Scene Diffusion From a Single RGB Image
  4. 2025SceneCompleter: Dense 3D Scene Completion for Generative Novel View Synthesis
  5. 2024MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation
  6. 2025Geometry-Aware Diffusion Models for Multiview Scene Inpainting
  7. 2023Breathing New Life into 3D Assets with Generative Repainting
  8. 2023Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion