mlsanityData to 7 Oct 2026

HoloDiffusion: Training a 3D Diffusion Model using 2D Images

arXiv 2303.16509 ↗First posted 29 Mar 2023cs.CV cs.GRcited by 106

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Diffusion models have emerged as the best approach for generative modeling of 2D images. Part of their success is due to the possibility of training them on millions if not billions of images with a stable learning objective. However, extending these models to 3D remains difficult for two reasons. First, finding a large quantity of 3D training data is much more complex than for 2D images. Second, while it is conceptually trivial to extend the models to operate on 3D rather than 2D grids, the associated cubic growth in memory and compute complexity makes this infeasible. We address the first challenge by introducing a new diffusion setup that can be trained, end-to-end, with only posed 2D images for supervision; and the second challenge by proposing an image formation model that decouples model memory from spatial memory. We evaluate our method on real-world data, using the CO3D dataset which has not been used to train 3D generative models before. We show that our diffusion models are scalable, train robustly, and are competitive in terms of sample quality and fidelity to existing approaches for 3D generative modeling.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 61 of its 73 references that are in the map. First posted on arXiv 29 Mar 2023.

Builds on

61 of 73 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  2. 2022Novel View Synthesis with Diffusion Models6 Oct
  3. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  4. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  8. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  9. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  10. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  11. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
  12. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,957
38 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,607
  3. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  4. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  5. 2022TensoRF: Tensorial Radiance Fieldscited by 1,275
  6. 2018Demystifying MMD GANscited by 1,268
  7. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  8. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  9. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 961
  10. 2020NeRF++: Analyzing and Improving Neural Radiance Fieldscited by 857
  11. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  12. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  13. 2021IBRNet: Learning Multi-View Image-Based Renderingcited by 698
  14. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  15. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  16. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  17. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 569
  18. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  19. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  20. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  21. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  22. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  23. 2022Understanding Diffusion Models: A Unified Perspectivecited by 337
  24. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  25. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  26. 2020State of the Art on Neural Renderingcited by 276
  27. 2019Deferred Neural Rendering: Image Synthesis using Neural Texturescited by 253
  28. 2021BANMo: Building Animatable 3D Neural Models from Many Casual Videoscited by 195
  29. 2019Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wildcited by 180
  30. 2019Neural Volumes: Learning Dynamic Renderable Volumes from Imagescited by 173
  31. 2018Escaping Plato's Cave: 3D Shape From Adversarial Renderingcited by 170
  32. 2020BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Imagescited by 166
  33. 2022GAUDI: A Neural Architect for Immersive 3D Scene Generationcited by 136
  34. 2022VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Gridscited by 133
  35. 2022Neural Wavelet-domain Diffusion for 3D Shape Generationcited by 132
  36. 2021Unconstrained Scene Generation with Locally Conditioned Radiance Fieldscited by 130
  37. 2020Self-supervised Single-view 3D Reconstruction via Semantic Consistencycited by 127
  38. 20163D Shape Induction from 2D Views of Multiple Objectscited by 121

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 34; 2024: 55; 2025: 10; 2026: 7.

106 papers.

Sort:

By area:

  1. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  2. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  3. 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
  4. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 298
  5. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  6. 2023FSGS: Real-Time Few-shot View Synthesis using Gaussian Splattingcited by 278
  7. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  8. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generationcited by 256
  9. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 213
  10. 2023Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyondcited by 139
  11. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  12. 2024MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesiscited by 123
38 more
  1. 2024latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstructioncited by 114
  2. 2023Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Datacited by 113
  3. 2023Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervisioncited by 112
  4. 2024Era3D: High-Resolution Multiview Diffusion using Efficient Row-wise Attentioncited by 99
  5. 2024V3D: Video Diffusion Models are Effective 3D Generatorscited by 88
  6. 2024DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splattingcited by 81
  7. 2023Free3D: Consistent Novel View Synthesis without 3D Representationcited by 72
  8. 20233D-aware Image Generation using 2D Diffusion Modelscited by 69
  9. 2024Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materialscited by 66
  10. 2024Advances in 3D Generation: A Surveycited by 62
  11. 2024SPAD : Spatially Aware Multiview Diffuserscited by 62
  12. 2024ViewDiff: 3D-Consistent Image Generation with Text-to-Image Modelscited by 57
  13. 2024MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Modelcited by 55
  14. 2024AGG: Amortized Generative 3D Gaussians for Single Image to 3Dcited by 51
  15. 2024TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusioncited by 51
  16. 2025DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generationcited by 49
  17. 2024Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Modelscited by 44
  18. 2023Text2Immersion: Generative Immersive Scene with 3D Gaussianscited by 43
  19. 2024EditSplat: Multi-View Fusion and Attention-Guided Optimization for View-Consistent 3D Scene Editing with 3D Gaussian Splattingcited by 40
  20. 2023Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstructioncited by 39
  21. 2023LLMR: Real-time Prompting of Interactive Worlds using Large Language Modelscited by 39
  22. 2023VolumeDiffusion: Flexible Text-to-3D Generation with Efficient Volumetric Encodercited by 39
  23. 2024MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesiscited by 35
  24. 2024SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Viewscited by 35
  25. 2023Disentangled Clothed Avatar Generation from Text Descriptionscited by 33
  26. 2023HoloFusion: Towards Photo-realistic 3D Generative Modelingcited by 32
  27. 2023GeoDream: Disentangling 2D and Geometric Priors for High-Fidelity and Consistent 3D Generationcited by 30
  28. 2023Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shapecited by 29
  29. 2023FreeDrag: Feature Dragging for Reliable Point-based Image Editingcited by 28
  30. 2024Recent Trends in 3D Reconstruction of General Non-Rigid Scenescited by 28
  31. 2024SuperGaussian: Repurposing Video Models for 3D Super Resolutioncited by 22
  32. 2024AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoisingcited by 22
  33. 2023Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Modelcited by 20
  34. 2023PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusioncited by 20
  35. 2023NViST: In the Wild New View Synthesis from a Single Image with Transformerscited by 19
  36. 2024Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabricationcited by 19
  37. 2024SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrixcited by 19
  38. 2024A Comprehensive Survey of Convolutions in Deep Learning: Applications, Challenges, and Future Trendscited by 16

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation
  2. 2022Efficient Diffusion Models for Vision: A Survey
  3. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practices
  4. 2022Let us Build Bridges: Understanding and Extending Diffusion Generative Models
  5. 2024Ouroboros3D: Image-to-3D Generation via 3D-aware Recursive Diffusion
  6. 2022Diffusion Models: A Comprehensive Survey of Methods and Applications
  7. 2023AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration
  8. 2024DiffuseHigh: Training-free Progressive High-Resolution Image Synthesis through Structure Guidance