mlsanityData to 7 Oct 2026

3D-aware Image Generation using 2D Diffusion Models

arXiv 2303.17905 ↗First posted 31 Mar 2023cs.CVcited by 69

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

In this paper, we introduce a novel 3D-aware image generation method that leverages 2D diffusion models. We formulate the 3D-aware image generation task as multiview 2D image set generation, and further to a sequential unconditional-conditional multiview image generation process. This allows us to utilize 2D diffusion models to boost the generative modeling power of the method. Additionally, we incorporate depth information from monocular depth estimators to construct the training data for the conditional diffusion model using only still images. We train our method on a large-scale dataset, i.e., ImageNet, which is not addressed by previous methods. It produces high-quality images that significantly outperform prior methods. Furthermore, our approach showcases its capability to generate instances with large view angles, even though the training images are diverse and unaligned, gathered from "in-the-wild" real-world environments.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 49 of its 61 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

49 of 61 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023HoloDiffusion: Training a 3D Diffusion Model using 2D Images29 Mar
  2. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  3. 2022RGBD2: Generative Scene Synthesis via Incremental View Inpainting using RGBD Diffusion Models12 Dec
  4. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusion2 Dec
  5. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  6. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  7. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  8. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation17 Nov
  9. 2022Novel View Synthesis with Diffusion Models6 Oct
  10. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan
  11. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 38

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  12. 2016Improved Techniques for Training GANscited by 2,938
26 more
  1. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  2. 2021NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstructioncited by 1,701
  3. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  4. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  5. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  6. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  7. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  8. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  9. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  10. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  11. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  12. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  13. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  14. 2021SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Modelscited by 415
  15. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  16. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 239
  17. 2021GRAM: Generative Radiance Manifolds for 3D-Aware Image Generationcited by 216
  18. 20193D Ken Burns Effect from a Single Imagecited by 128
  19. 2022EpiGRAF: Rethinking training of 3D GANscited by 108
  20. 2022InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Imagescited by 82
  21. 2022GRAM-HD: 3D-Consistent Image Generation at High Resolution with Generative Radiance Manifoldscited by 73
  22. 2022Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Imagescited by 59
  23. 20233D generation on ImageNetcited by 47
  24. 2022Self-Distilled StyleGAN: Towards Generation from Internet Photoscited by 37
  25. 20223D-Aware Indoor Scene Synthesis with Depth Priorscited by 32
  26. 2023VQ3D: Learning a 3D-Aware Generative Model on ImageNetcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 32; 2025: 17; 2026: 4.

69 papers.

Sort:

By area:

  1. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  2. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  3. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  4. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  5. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  6. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  7. 2023ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Imagecited by 109
  8. 2024DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generationcited by 93
  9. 2023Consistent123: Improve Consistency for One Image to 3D Object Synthesiscited by 84
  10. 2024NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizercited by 79
  11. 2025RayZer: A Self-supervised Large View Synthesis Modelcited by 74
  12. 2023EpiDiff: Enhancing Multi-View Synthesis via Localized Epipolar-Constrained Diffusioncited by 72
38 more
  1. 2024MultiDiff: Consistent Novel View Synthesis from a Single Imagecited by 67
  2. 2024Advances in 3D Generation: A Surveycited by 62
  3. 2024ViewDiff: 3D-Consistent Image Generation with Text-to-Image Modelscited by 57
  4. 2024Part123: Part-aware 3D Reconstruction from a Single-view Imagecited by 54
  5. 2023Single-Image 3D Human Digitization with Shape-Guided Diffusioncited by 42
  6. 2025GenFusion: Closing the Loop between Reconstruction and Generation via Videoscited by 41
  7. 2023Cascade-Zero123: One Image to Highly Consistent 3D with Self-Prompted Nearby Viewscited by 32
  8. 2025SynCity: Training-Free Generation of 3D Worldscited by 30
  9. 2024Real3D: Scaling Up Large Reconstruction Models with Real-World Imagescited by 24
  10. 2024Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D Diffusioncited by 24
  11. 2024Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpaintingcited by 23
  12. 2024Beyond Deepfake Images: Detecting AI-Generated Videoscited by 22
  13. 2024ConsistDreamer: 3D-Consistent 2D Diffusion for High-Fidelity Scene Editingcited by 18
  14. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 18
  15. 2023Controllable Group Choreography using Contrastive Diffusioncited by 17
  16. 2024PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preferencecited by 17
  17. 2024Denoising Diffusion via Image-Based Renderingcited by 15
  18. 2024GenesisTex: Adapting Image Denoising Diffusion to Texture Spacecited by 15
  19. 2024ViewFusion: Towards Multi-View Consistency via Interpolated Denoisingcited by 14
  20. 2023AI-Generated Images as Data Source: The Dawn of Synthetic Eracited by 13
  21. 2023Inpaint3D: 3D Scene Content Generation using 2D Inpainting Diffusioncited by 13
  22. 2024Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embeddingcited by 13
  23. 2025A Recipe for Generating 3D Worlds From a Single Imagecited by 13
  24. 2024Diffusion Models are Geometry Critics: Single Image 3D Editing Using Pre-Trained Diffusion Priorscited by 12
  25. 2024LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priorscited by 11
  26. 2023WildFusion: Learning 3D-Aware Latent Diffusion Models in View Spacecited by 8
  27. 2023MVControl: Adding Conditional Control to Multi-view Diffusion for Controllable Text-to-3D Generationcited by 8
  28. 2025Wonder3D++: Cross-domain Diffusion for High-fidelity 3D Generation from a Single Imagecited by 8
  29. 2024E3: Ensemble of Expert Embedders for Adapting Synthetic Image Detectors to New Generators Using Limited Datacited by 7
  30. 2024DreamCraft3D++: Efficient Hierarchical 3D Generation with Multi-Plane Reconstruction Modelcited by 7
  31. 2024Direct and Explicit 3D Generation from a Single Imagecited by 6
  32. 2025ART-DECO: Arbitrary Text Guidance for 3D Detailizer Constructioncited by 6
  33. 2025LACONIC: A 3D Layout Adapter for Controllable Image Creationcited by 6
  34. 2024G3DR: Generative 3D Reconstruction in ImageNetcited by 5
  35. 2025Self-attention-based Diffusion Model for Time-series Imputation in Partial Blackout Scenarioscited by 5
  36. 2025MV-Performer: Taming Video Diffusion Model for Faithful and Synchronized Multi-view Performer Synthesiscited by 4
  37. 2025F3D-Gaus: Feed-forward 3D-aware Generation on ImageNet with Cycle-Aggregative Gaussian Splattingcited by 3
  38. 2025DMS:Diffusion-Based Multi-Baseline Stereo Generation for Improving Self-Supervised Depth Estimationcited by 3

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy
  2. 2024Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion
  3. 2023MVDream: Multi-view Diffusion for 3D Generation
  4. 2024Diffusion2: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models
  5. 2023Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data
  6. 2024A Simple Approach to Unifying Diffusion-based Conditional Generation
  7. 2025Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation
  8. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion