mlsanityData to 7 Oct 2026

RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation

arXiv 2211.09869 ↗First posted 17 Nov 2022cs.CV cs.LGcited by 162

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Diffusion models currently achieve state-of-the-art performance for both conditional and unconditional image generation. However, so far, image diffusion models do not support tasks required for 3D understanding, such as view-consistent 3D generation or single-view object reconstruction. In this paper, we present RenderDiffusion, the first diffusion model for 3D generation and inference, trained using only monocular 2D supervision. Central to our method is a novel image denoising architecture that generates and renders an intermediate three-dimensional representation of a scene in each denoising step. This enforces a strong inductive structure within the diffusion process, providing a 3D consistent representation while only requiring 2D supervision. The resulting 3D representation can be rendered from any view. We evaluate RenderDiffusion on FFHQ, AFHQ, ShapeNet and CLEVR datasets, showing competitive performance for generation of 3D scenes and inference of 3D scenes from 2D images. Additionally, our diffusion-based approach allows us to use 2D inpainting to edit 3D scenes.

Key facts. Cited by 162 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 57 of its 75 references that are in the map. First posted on arXiv 17 Nov 2022.

Builds on

57 of 75 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Novel View Synthesis with Diffusion Models6 Oct
  2. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  3. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan
  4. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 53

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
38 more
  1. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  2. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  3. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  4. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  5. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  6. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  7. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  10. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  11. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  12. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  13. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  14. 2021Variational Diffusion Modelscited by 1,145
  15. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  16. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  17. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  18. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  19. 2016Pixel Recurrent Neural Networkscited by 814
  20. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  21. 2020Convolutional Occupancy Networkscited by 760
  22. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 750
  23. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 735
  24. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  25. 2020Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humanscited by 634
  26. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  27. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  28. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  29. 2022Point-NeRF: Point-based Neural Radiance Fieldscited by 503
  30. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  31. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  32. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  33. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  34. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  35. 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
  36. 2020BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Imagescited by 166
  37. 2022ARF: Artistic Radiance Fieldscited by 147
  38. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 54; 2024: 63; 2025: 32; 2026: 10.

162 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  3. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  4. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  5. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 359
  6. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  7. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  8. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  9. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 213
  10. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusioncited by 204
  11. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstructioncited by 180
  12. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 179
38 more
  1. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  2. 2024MeshLRM: Large Reconstruction Model for High-Quality Meshescited by 138
  3. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  4. 2024latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstructioncited by 114
  5. 2023Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Datacited by 113
  6. 2024Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Imagecited by 113
  7. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  8. 20244Diffusion: Multi-view Video Diffusion Model for 4D Generationcited by 104
  9. 2024Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attentioncited by 99
  10. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 91
  11. 2023AVID: Any-Length Video Inpainting with Diffusion Modelcited by 81
  12. 2025PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformerscited by 78
  13. 2023Long-Term Photometric Consistent Novel View Synthesis with Diffusion Modelscited by 76
  14. 2023SKED: Sketch-guided Text-based 3D Editingcited by 72
  15. 2023Free3D: Consistent Novel View Synthesis without 3D Representationcited by 72
  16. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
  17. 20233D-aware Image Generation using 2D Diffusion Modelscited by 69
  18. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illuminationcited by 67
  19. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 67
  20. 2023Texture Generation on 3D Meshes with Point-UV Diffusioncited by 63
  21. 2024ViewDiff: 3D-Consistent Image Generation with Text-to-Image Modelscited by 57
  22. 2025Bolt3D: Generating 3D Scenes in Secondscited by 57
  23. 2023RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texturecited by 53
  24. 2024SemCity: Semantic Scene Generation with Triplane Diffusioncited by 53
  25. 2025Avat3r: Large Animatable Gaussian Reconstruction Model for High-fidelity 3D Head Avatarscited by 52
  26. 2025Amodal3R: Amodal 3D Reconstruction from Occluded 2D Imagescited by 52
  27. 2022DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Modelscited by 51
  28. 2025DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generationcited by 49
  29. 2025SceneGen: Single-Image 3D Scene Generation in One Feedforward Passcited by 49
  30. 2024Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Modelscited by 44
  31. 2024EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splattingcited by 40
  32. 2024Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusioncited by 39
  33. 2023PrimDiffusion: Volumetric Primitives Diffusion for 3D Human Generationcited by 33
  34. 2023HoloFusion: Towards Photo-realistic 3D Generative Modelingcited by 32
  35. 2024Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation and Reconstructioncited by 32
  36. 2023DreamSparse: Escaping from Plato's Cave with 2D Frozen Diffusion Model Given Sparse Viewscited by 31
  37. 2023Controllable Mind Visual Diffusion Modelcited by 29
  38. 2023Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shapecited by 29

50 most cited of 162 All 162 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20243D-Consistent Image Inpainting with Diffusion Models
  2. 2022TetraDiffusion: Tetrahedral Diffusion Models for 3D Shape Generation
  3. 2023HoloDiffusion: Training a 3D Diffusion Model using 2D Images
  4. 2024Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion
  5. 2024Coherent 3D Scene Diffusion From a Single RGB Image
  6. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis
  7. 2024DSplats: 3D Generation by Denoising Splats-Based Multiview Diffusion Models
  8. 2024Move Anything with Layered Scene Diffusion