mlsanityData to 14 Oct 2026

Challenges and Opportunities in 3D Content Generation

arXiv 2405.15335 ↗First posted 24 May 2024cs.GRcited by 1

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

This paper explores the burgeoning field of 3D content generation within the landscape of Artificial Intelligence Generated Content (AIGC) and large-scale models. It investigates innovative methods like Text-to-3D and Image-to-3D, which translate text or images into 3D objects, reshaping our understanding of virtual and real-world simulations. Despite significant advancements in text and image generation, automatic 3D content generation remains nascent. This paper emphasizes the urgency for further research in this area. By leveraging pre-trained diffusion models, which have demonstrated prowess in high-fidelity image generation, this paper aims to summary 3D content creation, addressing challenges such as data scarcity and computational resource limitations. Additionally, this paper discusses the challenges and proposes solutions for using pre-trained diffusion models in 3D content generation. By synthesizing relevant research and outlining future directions, this study contributes to advancing the field of 3D content generation amidst the proliferation of large-scale AIGC models.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 67 of its 75 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

67 of 75 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024ART3D: 3D Gaussian Splatting for Text-Guided Artistic Scenes Generation17 May
  2. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior25 Oct
  3. 2023Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model23 Oct
  4. 2023HiFi-123: Towards High-fidelity One Image to 3D Content Generation10 Oct
  5. 2023SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D4 Oct
  6. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors30 Jun
  7. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  8. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Models5 Apr
  9. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior24 Mar
  10. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  11. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  12. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors6 Dec
4 more
  1. 2022NeuralLift-360: Lifting An In-the-wild 2D Photo to A 3D Object with 360deg Views29 Nov
  2. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  3. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
  4. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  9. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  10. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  11. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  12. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,964
38 more
  1. 2021Plenoxels: Radiance Fields without Neural Networkscited by 1,613
  2. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
  3. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  4. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 964
  5. 2020Neural Sparse Voxel Fieldscited by 959
  6. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
  7. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  8. 2021MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereocited by 740
  9. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  10. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 647
  11. 2020AdaBins: Depth Estimation using Adaptive Binscited by 601
  12. 2021Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fieldscited by 575
  13. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  14. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  15. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  16. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 460
  17. 20193D Packing for Self-Supervised Monocular Depth Estimationcited by 439
  18. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  19. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  20. 2019Neural Point-Based Graphicscited by 327
  21. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
  22. 2023Text-to-3D using Gaussian Splattingcited by 299
  23. 2017Grammar Variational Autoencodercited by 285
  24. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  25. 2020Learning Gradient Fields for Shape Generationcited by 245
  26. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  27. 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
  28. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 199
  29. 2021Point-Based Neural Rendering with Per-View Optimizationcited by 190
  30. 2016Variational Autoencoder for Deep Learning of Images, Labels and Captionscited by 137
  31. 2022VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Gridscited by 133
  32. 2023HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splattingcited by 89
  33. 2023Gaussian Shell Maps for Efficient 3D Human Generationcited by 61
  34. 2023LayerDiffusion: Layered Controlled Image Editing with Diffusion Modelscited by 40
  35. 2022NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fieldscited by 33
  36. 2022High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priorscited by 28
  37. 2024Tuning-Free Image Customization with Image and Text Guidancecited by 25
  38. 2024Sketch-to-Architecture: Generative AI-aided Architectural Designcited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Unifi3D: A Study on 3D Representations for Generation and Reconstruction in a Common Frameworkcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey On Text-to-3D Contents Generation In The Wild
  2. 2023Direct2.5: Diverse Text-to-3D Generation via Multi-view 2.5D Diffusion
  3. 2023EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior
  4. 2023PI3D: Efficient Text-to-3D Generation with Pseudo-Image Diffusion
  5. 2023DreamTime: An Improved Optimization Strategy for Diffusion-Guided 3D Generation
  6. 2023X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generation
  7. 2025Recent Advances in 3D Object and Scene Generation: A Survey
  8. 2023IT3D: Improved Text-to-3D Generation with Explicit View Synthesis