mlsanityData to 14 Oct 2026

Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction

arXiv 2304.06714 ↗First posted 13 Apr 2023cs.CVcited by 180

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images. Despite numerous task-specific methods, developing a comprehensive model remains challenging. In this paper, we present SSDNeRF, a unified approach that employs an expressive diffusion model to learn a generalizable prior of neural radiance fields (NeRF) from multi-view images of diverse objects. Previous studies have used two-stage approaches that rely on pretrained NeRFs as real data to train diffusion models. In contrast, we propose a new single-stage training paradigm with an end-to-end objective that jointly optimizes a NeRF auto-decoder and a latent diffusion model, enabling simultaneous 3D reconstruction and prior learning, even from sparsely available views. At test time, we can directly sample the diffusion prior for unconditional generation, or combine it with arbitrary observations of unseen objects for NeRF reconstruction. SSDNeRF demonstrates robust results comparable to or better than leading task-specific methods in unconditional generation and single/sparse-view 3D reconstruction.

Key facts. Cited by 180 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 51 of its 66 references that are in the map. First posted on arXiv 13 Apr 2023.

Builds on

51 of 66 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  2. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors6 Dec
  3. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusion2 Dec
  4. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  5. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation17 Nov
  6. 2022Novel View Synthesis with Diffusion Models6 Oct
  7. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  8. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 43

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2013Auto-Encoding Variational Bayescited by 4,053
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  10. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  11. 2022Video Diffusion Modelscited by 2,305
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
31 more
  1. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  2. 2018Demystifying MMD GANscited by 1,293
  3. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  4. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  5. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  6. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 740
  7. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  8. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 689
  9. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 672
  10. 2021Score-based Generative Modeling in Latent Spacecited by 636
  11. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  12. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 449
  13. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  14. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
  15. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  16. 2022From data to functa: Your data point is a function and you can treat it like onecited by 187
  17. 2021CodeNeRF: Disentangled Neural Radiance Fields for Object Categoriescited by 183
  18. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 180
  19. 2022Generative Modelling With Inverse Heat Dissipationcited by 167
  20. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  21. 2022VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Gridscited by 133
  22. 2021Unconstrained Scene Generation with Locally Conditioned Radiance Fieldscited by 130
  23. 2022EpiGRAF: Rethinking training of 3D GANscited by 108
  24. 2019Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesiscited by 104
  25. 2022Vision Transformer for NeRF-Based View Synthesis from a Single Input Imagecited by 104
  26. 2022NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstructioncited by 100
  27. 2021LOLNeRF: Learn from One Lookcited by 84
  28. 2019HoloGAN: Unsupervised learning of 3D representations from natural imagescited by 59
  29. 2023Factor Fields: A Unified Framework for Neural Fields and Beyondcited by 58
  30. 2021Diffusion Priors In Variational Autoencoderscited by 25
  31. 2021MonoCInIS: Camera Independent Monocular 3D Object Detection using Instance Segmentationcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 40; 2024: 85; 2025: 30; 2026: 25.

180 papers.

Sort:

By area:

  1. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  2. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 770
  3. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  4. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 689
  5. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 645
  6. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 378
  7. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 364
  8. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 301
  9. 2023Text-to-3D using Gaussian Splattingcited by 299
  10. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 231
  11. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 215
  12. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 214
38 more
  1. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  2. 2024MeshLRM: Large Reconstruction Model for High-Quality Meshescited by 139
  3. 2023State of the Art on Diffusion Models for Visual Computingcited by 127
  4. 2024latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstructioncited by 118
  5. 2023HiFA: High-fidelity Text-to-3D Generation with Advanced Diffusion Guidancecited by 110
  6. 2025VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memorycited by 109
  7. 20243DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusioncited by 101
  8. 2023BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrappingcited by 98
  9. 2024EscherNet: A Generative Model for Scalable View Synthesiscited by 95
  10. 2025MEt3R: Measuring Multi-View Consistency in Generated Imagescited by 95
  11. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 92
  12. 2023HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splattingcited by 89
  13. 2024EdgeRunner: Auto-regressive Auto-encoder for Artistic Mesh Generationcited by 87
  14. 2024Classifier-Free Guidance is a Predictor-Correctorcited by 84
  15. 2024BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolationcited by 83
  16. 2024NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizercited by 79
  17. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
  18. 2023LooseControl: Lifting ControlNet for Generalized Depth Conditioningcited by 68
  19. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illuminationcited by 67
  20. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  21. 2024Advances in 3D Generation: A Surveycited by 63
  22. 2024Part123: Part-aware 3D Reconstruction from a Single-view Imagecited by 55
  23. 2024LT3SD: Latent Trees for 3D Scene Diffusioncited by 55
  24. 2024AGG: Amortized Generative 3D Gaussians for Single Image to 3Dcited by 53
  25. 2024SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstructioncited by 52
  26. 2024DiffGS: Functional Gaussian Splatting Diffusioncited by 51
  27. 2023AutoDecoding Latent 3D Diffusion Modelscited by 50
  28. 2023Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusioncited by 50
  29. 2025Efficient Part-level 3D Object Generation via Dual Volume Packingcited by 48
  30. 2024Generic 3D Diffusion Adapter Using Controlled Multi-View Editingcited by 47
  31. 2024GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modelingcited by 45
  32. 2024L3DG: Latent 3D Gaussian Diffusioncited by 42
  33. 2025SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Imagescited by 42
  34. 2025GenFusion: Closing the Loop between Reconstruction and Generation via Videoscited by 41
  35. 2023Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstructioncited by 39
  36. 2024Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusioncited by 39
  37. 2023Alchemist: Parametric Control of Material Properties with Diffusion Modelscited by 38
  38. 2023Tracking and Mapping in Medical Computer Vision: A Reviewcited by 36

50 most cited of 180 All 180 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ID-NeRF: Indirect Diffusion-guided Neural Radiance Fields for Generalizable View Synthesis
  2. 2025DT-NeRF: A Diffusion and Transformer-Based Optimization Approach for Neural Radiance Fields in 3D Reconstruction
  3. 2023ReconFusion: 3D Reconstruction with Diffusion Priors
  4. 2023Learning a Diffusion Prior for NeRFs
  5. 2024Taming Latent Diffusion Model for Neural Radiance Field Inpainting
  6. 2023Sparse3D: Distilling Multiview-Consistent Diffusion for Object Reconstruction from Sparse Views
  7. 2023Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation
  8. 2024Diffusion Priors for Dynamic View Synthesis from Monocular Videos