mlsanityData to 7 Oct 2026

NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors

arXiv 2212.03267 ↗First posted 6 Dec 2022cs.CVcited by 156

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

2D-to-3D reconstruction is an ill-posed problem, yet humans are good at solving this problem due to their prior knowledge of the 3D world developed over years. Driven by this observation, we propose NeRDi, a single-view NeRF synthesis framework with general image priors from 2D diffusion models. Formulating single-view reconstruction as an image-conditioned 3D generation problem, we optimize the NeRF representations by minimizing a diffusion loss on its arbitrary view renderings with a pretrained image diffusion model under the input-view constraint. We leverage off-the-shelf vision-language models and introduce a two-section language guidance as conditioning inputs to the diffusion model. This is essentially helpful for improving multiview content coherence as it narrows down the general image prior conditioned on the semantic and visual features of the single-view input image. Additionally, we introduce a geometric loss based on estimated depth maps to regularize the underlying 3D geometry of the NeRF. Experimental results on the DTU MVS dataset show that our method can synthesize novel views with higher quality even compared to existing methods trained on this dataset. We also demonstrate our generalizability in zero-shot NeRF synthesis for in-the-wild images.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 46 of its 61 references that are in the map. First posted on arXiv 6 Dec 2022.

Builds on

46 of 61 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Novel View Synthesis with Diffusion Models6 Oct
  2. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  3. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
31 more
  1. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2021Vision Transformers for Dense Predictioncited by 1,789
  4. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  6. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  7. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  8. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  9. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 735
  10. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 732
  11. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  12. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  13. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  14. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  15. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  16. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  17. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  18. 2021Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesiscited by 464
  19. 2021Dense Depth Priors for Neural Radiance Fields from Sparse Input Viewscited by 301
  20. 2021Neural Rays for Occlusion-aware Image-based Renderingcited by 192
  21. 2021Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenescited by 190
  22. 2020GRF: Learning a General Radiance Field for 3D Representation and Renderingcited by 183
  23. 2022SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Imagecited by 165
  24. 2021NeRF--: Neural Radiance Fields Without Known Camera Parameterscited by 155
  25. 2021GNeRF: GAN-based Neural Radiance Field without Posed Cameracited by 153
  26. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  27. 2022Highly Accurate Dichotomous Image Segmentationcited by 110
  28. 2021Shelf-Supervised Mesh Prediction in the Wildcited by 65
  29. 2020Equivariant Neural Renderingcited by 47
  30. 2022Pre-train, Self-train, Distill: A simple recipe for Supersizing 3D Reconstructioncited by 26
  31. 2022im2nerf: Image to Neural Radiance Field in the Wildcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 63; 2024: 76; 2025: 14; 2026: 3.

156 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  3. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  4. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  5. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  6. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  7. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  8. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  9. 2023FSGS: Real-Time Few-shot View Synthesis using Gaussian Splattingcited by 278
  10. 2024DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalizationcited by 256
  11. 2023SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesiscited by 247
  12. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
38 more
  1. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstructioncited by 180
  2. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 179
  3. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  4. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  5. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  6. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  7. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  8. 2023Consistent4D: Consistent 360deg Dynamic Object Generation from Monocular Videocited by 124
  9. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistencycited by 117
  10. 2023ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Imagecited by 109
  11. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusioncited by 109
  12. 2023TeCH: Text-guided Reconstruction of Lifelike Clothed Humanscited by 104
  13. 2024EscherNet: A Generative Model for Scalable View Synthesiscited by 94
  14. 2023TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Modelscited by 86
  15. 2024DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splattingcited by 81
  16. 2024TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Promptscited by 76
  17. 2023Diffusion-Guided Reconstruction of Everyday Hand-Object Interaction Clipscited by 73
  18. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
  19. 2024GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warpingcited by 72
  20. 2024IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illuminationcited by 67
  21. 2024Advances in 3D Generation: A Surveycited by 62
  22. 2024SPAD : Spatially Aware Multiview Diffuserscited by 62
  23. 2024Learning the 3D Fauna of the Webcited by 58
  24. 2024MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Modelcited by 55
  25. 2023RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent Geometry and Texturecited by 53
  26. 2024SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstructioncited by 52
  27. 2024AGG: Amortized Generative 3D Gaussians for Single Image to 3Dcited by 51
  28. 2023DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Imagecited by 48
  29. 2023A Comprehensive Survey on Knowledge Distillation of Diffusion Modelscited by 45
  30. 2024PhyRecon: Physically Plausible Neural Scene Reconstructioncited by 44
  31. 2024G-HOP: Generative Hand-Object Prior for Interaction Reconstruction and Grasp Synthesiscited by 43
  32. 2025SPAR3D: Stable Point-Aware Reconstruction of 3D Objects from Single Imagescited by 41
  33. 2024OctFusion: Octree-based Diffusion Models for 3D Shape Generationcited by 40
  34. 2023Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstructioncited by 39
  35. 2024MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generationcited by 35
  36. 2024SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Viewscited by 35
  37. 2023HiFi-123: Towards High-fidelity One Image to 3D Content Generationcited by 34
  38. 2023HoloFusion: Towards Photo-realistic 3D Generative Modelingcited by 32

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Model
  2. 2024Generative Lifting of Multiview to 3D from Unknown Pose: Wrapping NeRF inside Diffusion
  3. 2024ID-NeRF: Indirect Diffusion-guided Neural Radiance Fields for Generalizable View Synthesis
  4. 2023ReconFusion: 3D Reconstruction with Diffusion Priors
  5. 2023Efficient-3DiM: Learning a Generalizable Single-image Novel-view Synthesizer in One Day
  6. 2025DT-NVS: Diffusion Transformers for Novel View Synthesis
  7. 2023Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning
  8. 2024DiSR-NeRF: Diffusion-Guided View-Consistent Super-Resolution NeRF