mlsanityData to 14 Oct 2026

Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusion

arXiv 2312.03869 ↗First posted 6 Dec 2023cs.CVcited by 13

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

This paper presents a novel approach to inpainting 3D regions of a scene, given masked multi-view images, by distilling a 2D diffusion model into a learned 3D scene representation (e.g. a NeRF). Unlike 3D generative methods that explicitly condition the diffusion model on camera pose or multi-view information, our diffusion model is conditioned only on a single masked 2D image. Nevertheless, we show that this 2D diffusion model can still serve as a generative prior in a 3D multi-view reconstruction problem where we optimize a NeRF using a combination of score distillation sampling and NeRF reconstruction losses. Predicted depth is used as additional supervision to encourage accurate geometry. We compare our approach to 3D inpainting methods that focus on object removal. Because our method can generate content to fill any 3D masked region, we additionally demonstrate 3D object completion, 3D object replacement, and 3D scene completion.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 55 of its 65 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

55 of 65 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023DreamSparse: Escaping from Plato's Cave with 2D Frozen Diffusion Model Given Sparse Views6 Jun
  2. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  3. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Models5 Apr
  4. 20233D-aware Image Generation using 2D Diffusion Models31 Mar
  5. 2023Consistent View Synthesis with Pose-Guided Diffusion Models30 Mar
  6. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  7. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  8. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusion2 Dec
  9. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  10. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  11. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  12. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation17 Nov
3 more
  1. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  2. 2022Novel View Synthesis with Diffusion Models6 Oct
  3. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 40

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2023Segment Anythingcited by 8,234
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  11. 2022Video Diffusion Modelscited by 2,305
  12. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
28 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  2. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  3. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  4. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,299
  5. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,232
  6. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  7. 2021Variational Diffusion Modelscited by 1,153
  8. 2022On Distillation of Guided Diffusion Modelscited by 725
  9. 2023Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fieldscited by 671
  10. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 615
  11. 2018Free-Form Image Inpainting with Gated Convolutioncited by 594
  12. 2018Generative Image Inpainting with Contextual Attentioncited by 580
  13. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  14. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  15. 2023Instruct-NeRF2NeRF: Editing 3D Scenes with Instructionscited by 484
  16. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  17. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 180
  18. 2022Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wildcited by 160
  19. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  20. 2022SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fieldscited by 144
  21. 2023Noise-Free Score Distillationcited by 99
  22. 2023Nerfbusters: Removing Ghostly Artifacts from Casually Captured NeRFscited by 76
  23. 2022Removing Objects From Neural Radiance Fieldscited by 66
  24. 2023Reference-guided Controllable Inpainting of Neural Radiance Fieldscited by 40
  25. 2023OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fieldscited by 34
  26. 2023InNeRF360: Text-Guided 3D-Consistent Object Inpainting on 360-degree Neural Radiance Fieldscited by 16
  27. 2023Clutter Detection and Removal in 3D Scenes with View-Consistent Inpaintingcited by 9
  28. 2023RIC: Rotate-Inpaint-Complete for Generalizable Scene Reconstructioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 1.

13 papers.

Sort:

By area:

  1. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 41
  2. 2025AuraFusion360: Augmented Unseen Region Alignment for Reference-based 360deg Unbounded Scene Inpaintingcited by 21
  3. 2024Taming Latent Diffusion Model for Neural Radiance Field Inpaintingcited by 20
  4. 2025Orchid: Image Latent Diffusion for Joint Appearance and Geometry Generationcited by 17
  5. 2025Geometry-Aware Diffusion Models for Multiview Scene Inpaintingcited by 6
  6. 20243D StreetUnveiler with Semantic-aware 2DGS -- a simple baselinecited by 4
  7. 2025Native 3D Editing with Full Attentioncited by 4
  8. 2025SplatFill: 3D Scene Inpainting via Depth-Guided Gaussian Splattingcited by 3
  9. 2025Perspective-aware 3D Gaussian Inpainting with Multi-view Consistencycited by 3
  10. 2025ROODI: Reconstructing Occluded Objects with Denoising Inpainterscited by 1
  11. 2024Automatic occlusion removal from 3D maps for maritime situational awarenesscited by 0
  12. 2025Defurnishing with X-Ray Vision: Joint Removal of Furniture from Panoramas and Meshcited by 0
1 more
  1. 2026CoGeo-GS: Concept-Driven and Geometry-Aware Multi-Object Removal in 3D Scenescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023NeRFiller: Completing Scenes via Generative 3D Inpainting
  2. 2024RefFusion: Reference Adapted Diffusion Models for 3D Scene Inpainting
  3. 20243D-Consistent Image Inpainting with Diffusion Models
  4. 2026Benchmarking Single-Step Inpainting Methods for Multi-Object 3D Gaussian Splatting Scenes
  5. 2023iNVS: Repurposing Diffusion Inpainters for Novel View Synthesis
  6. 2024InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior
  7. 2024MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior
  8. 2023Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models