mlsanityData to 7 Oct 2026

Generative Novel View Synthesis with 3D-Aware Diffusion Models

arXiv 2304.02602 ↗First posted 5 Apr 2023cs.CV cs.AI cs.GRcited by 273

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We present a diffusion-based model for 3D-aware generative novel view synthesis from as few as a single input image. Our model samples from the distribution of possible renderings consistent with the input and, even in the presence of ambiguity, is capable of rendering diverse and plausible novel views. To achieve this, our method makes use of existing 2D diffusion backbones but, crucially, incorporates geometry priors in the form of a 3D feature volume. This latent feature field captures the distribution over possible scene representations and improves our method's ability to generate view-consistent novel renderings. In addition to generating novel views, our method has the ability to autoregressively synthesize 3D-consistent sequences. We demonstrate state-of-the-art results on synthetic renderings and room-scale scenes; we also show compelling results for challenging, real-world objects.

Key facts. Cited by 273 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 91 of its 119 references that are in the map. First posted on arXiv 5 Apr 2023.

Builds on

91 of 119 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Models3 Apr
  2. 20233D-aware Image Generation using 2D Diffusion Models31 Mar
  3. 2023Consistent View Synthesis with Pose-Guided Diffusion Models30 Mar
  4. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior24 Mar
  5. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  6. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  7. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  8. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  9. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  10. 2022Novel View Synthesis with Diffusion Models6 Oct
  11. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 80

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2022Generative Adversarial Networkscited by 5,603
38 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  2. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  3. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  6. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
  7. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  8. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  9. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  10. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  11. 2018Demystifying MMD GANscited by 1,268
  12. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  13. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  14. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  15. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  16. 2016Colorful Image Colorizationcited by 1,113
  17. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  18. 2019PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitizationcited by 818
  19. 2020Image Quality Assessment: Unifying Structure and Texture Similaritycited by 792
  20. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  21. 2020Convolutional Occupancy Networkscited by 760
  22. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 735
  23. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  24. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  25. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  26. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  27. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  28. 2018DeepVoxels: Learning Persistent 3D Feature Embeddingscited by 401
  29. 2021Advances in Neural Renderingcited by 365
  30. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  31. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  32. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  33. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  34. 2020Single-View View Synthesis with Multiplane Imagescited by 263
  35. 2019DeepView: View Synthesis with Learned Gradient Descentcited by 253
  36. 2020Learned Initializations for Optimizing Coordinate-Based Neural Representationscited by 240
  37. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 239
  38. 2020Free View Synthesiscited by 231

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 71; 2024: 118; 2025: 56; 2026: 28.

273 papers.

Sort:

By area:

  1. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  2. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  3. 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 668
  4. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  5. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  6. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  7. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  8. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  9. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 359
  10. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
  11. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 298
  12. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
38 more
  1. 2024Navigation World Modelscited by 262
  2. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  3. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generationcited by 256
  4. 2024TripoSR: Fast 3D Object Reconstruction from a Single Imagecited by 227
  5. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 213
  6. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
  7. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  8. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  9. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  10. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
  11. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  12. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  13. 2024ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Modelcited by 139
  14. 2023SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3Dcited by 134
  15. 2023DiffusionSat: A Generative Foundation Model for Satellite Imagerycited by 130
  16. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  17. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  18. 2024latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstructioncited by 114
  19. 2023Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Datacited by 113
  20. 2024Flash3D: Feed-Forward Generalisable 3D Scene Reconstruction from a Single Imagecited by 113
  21. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  22. 2024MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Viewscited by 111
  23. 2023TADA! Text to Animatable Digital Avatarscited by 109
  24. 2023ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Imagecited by 109
  25. 2023SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillationcited by 109
  26. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusioncited by 109
  27. 2023TeCH: Text-guided Reconstruction of Lifelike Clothed Humanscited by 104
  28. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 98
  29. 2024LN3DIFF++: Scalable Latent Neural Fields Diffusion for Speedy 3D Generationcited by 91
  30. 2025MEt3R: Measuring Multi-View Consistency in Generated Imagescited by 91
  31. 2024V3D: Video Diffusion Models are Effective 3D Generatorscited by 88
  32. 2023Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Modelscited by 79
  33. 2024NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizercited by 79
  34. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 76
  35. 2023Free3D: Consistent Novel View Synthesis without 3D Representationcited by 72
  36. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
  37. 2024GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warpingcited by 72
  38. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 67

50 most cited of 273 All 273 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Efficient-3DiM: Learning a Generalizable Single-image Novel-view Synthesizer in One Day
  2. 2026GenNVS: Geometry-enhanced Novel View Synthesis via Disentangled 3D Prior
  3. 2026Repurposing Geometric Foundation Models for Multi-view Diffusion
  4. 2025AR-1-to-3: Single Image to Consistent 3D Object Generation via Next-View Prediction
  5. 2024Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy
  6. 20243D-Adapter: Geometry-Consistent Multi-View Diffusion for High-Quality 3D Generation
  7. 2025DT-NVS: Diffusion Transformers for Novel View Synthesis
  8. 2023Consistent123: Improve Consistency for One Image to 3D Object Synthesis