mlsanityData to 7 Oct 2026

Novel View Synthesis with Diffusion Models

arXiv 2210.04628 ↗First posted 6 Oct 2022cs.CV cs.GR cs.LGcited by 299

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We present 3DiM, a diffusion model for 3D novel view synthesis, which is able to translate a single input view into consistent and sharp completions across many views. The core component of 3DiM is a pose-conditional image-to-image diffusion model, which takes a source view and its pose as inputs, and generates a novel view for a target pose as output. 3DiM can generate multiple views that are 3D consistent using a novel technique called stochastic conditioning. The output views are generated autoregressively, and during the generation of each novel view, one selects a random conditioning view from the set of available views at each denoising step. We demonstrate that stochastic conditioning significantly improves the 3D consistency of a naive sampler for an image-to-image diffusion model, which involves conditioning on a single fixed view. We compare 3DiM to prior work on the SRN ShapeNet dataset, demonstrating that 3DiM's generated completions from a single view achieve much higher fidelity, while being approximately 3D consistent. We also introduce a new evaluation methodology, 3D consistency scoring, to measure the 3D consistency of a generated object by training a neural field on the model's output views. 3DiM is geometry free, does not rely on hyper-networks or test-time optimization for novel view synthesis, and allows a single model to easily scale to a large number of scenes.

Key facts. Cited by 299 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 37 of its 57 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

37 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  2. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
23 more
  1. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  2. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  3. 2022Video Diffusion Modelscited by 2,280
  4. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  5. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  6. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  7. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  8. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  9. 2021Variational Diffusion Modelscited by 1,145
  10. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  11. 2018Which Training Methods for GANs do actually Converge?cited by 672
  12. 2021Maximum Likelihood Training of Score-Based Diffusion Modelscited by 670
  13. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  14. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  15. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  16. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  17. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 222
  18. 2021CodeNeRF: Disentangled Neural Radiance Fields for Object Categoriescited by 183
  19. 2022Vision Transformer for NeRF-Based View Synthesis from a Single Input Imagecited by 104
  20. 2021LOLNeRF: Learn from One Lookcited by 84
  21. 2017Transformation-Grounded Image Generation Network for Novel 3D View Synthesiscited by 59
  22. 2020Equivariant Neural Renderingcited by 47
  23. 2022A Diffusion Model Predicts 3D Shapes from 2D Microscopy Imagescited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 16; 2023: 111; 2024: 98; 2025: 52; 2026: 22.

299 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  3. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  4. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  5. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  6. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  7. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  8. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  9. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  10. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  11. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  12. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
38 more
  1. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 359
  2. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  3. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
  4. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusioncited by 309
  5. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  6. 2023Understanding Diffusion Objectives as the ELBO with Simple Data Augmentationcited by 293
  7. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  8. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generationcited by 256
  9. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  10. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 221
  11. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  12. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstructioncited by 180
  13. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 179
  14. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 178
  15. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  16. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  17. 2025History-Guided Video Diffusioncited by 165
  18. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generationcited by 162
  19. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  20. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priorscited by 156
  21. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  22. 2023TryOnDiffusion: A Tale of Two UNetscited by 141
  23. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  24. 2023SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3Dcited by 134
  25. 2022Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functionscited by 133
  26. 2024Cameras as Rays: Pose Estimation via Ray Diffusioncited by 129
  27. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  28. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 126
  29. 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
  30. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistencycited by 117
  31. 2024latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstructioncited by 114
  32. 2023Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Datacited by 113
  33. 2024Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Imagecited by 113
  34. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  35. 2024MV-Adapter: Multi-view Consistent Image Generation Made Easycited by 111
  36. 2023TADA! Text to Animatable Digital Avatarscited by 109
  37. 2023ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Imagecited by 109
  38. 2023SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillationcited by 109

50 most cited of 299 All 299 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Ctrl123: Consistent Novel View Synthesis via Closed-Loop Transcription
  2. 2024Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy
  3. 2025ConsistentDreamer: View-Consistent Meshes Through Balanced Multi-View Gaussian Optimization
  4. 2025CausNVS: Autoregressive Multi-view Diffusion for Flexible 3D Novel View Synthesis
  5. 2024Sculpt3D: Multi-View Consistent Text-to-3D Generation with Sparse 3D Prior
  6. 2024Towards High-Fidelity 3D Portrait Generation with Rich Details by Cross-View Prior-Aware Diffusion
  7. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model
  8. 2024Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion