mlsanityData to 14 Oct 2026

Neural Point Cloud Diffusion for Disentangled 3D Shape and Appearance Generation

arXiv 2312.14124 ↗First posted 21 Dec 2023cs.CVcited by 11

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Controllable generation of 3D assets is important for many practical applications like content creation in movies, games and engineering, as well as in AR/VR. Recently, diffusion models have shown remarkable results in generation quality of 3D objects. However, none of the existing models enable disentangled generation to control the shape and appearance separately. For the first time, we present a suitable representation for 3D diffusion models to enable such disentanglement by introducing a hybrid point cloud and neural radiance field approach. We model a diffusion process over point positions jointly with a high-dimensional feature space for a local density and radiance decoder. While the point positions represent the coarse shape of the object, the point features allow modeling the geometry and appearance details. This disentanglement enables us to sample both independently and therefore to control both separately. Our approach sets a new state of the art in generation compared to previous disentanglement-capable methods by reduced FID scores of 30-90% and is on-par with other non disentanglement-capable state-of-the art methods.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 53 of its 60 references that are in the map. First posted on arXiv 21 Dec 2023.

Builds on

53 of 60 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models20 Oct
  2. 2023Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction13 Apr
  3. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Models20 Mar
  4. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  5. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models26 Jan
  6. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  7. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusion2 Dec
  8. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  9. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  10. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation17 Nov
  11. 2022LION: Latent Point Diffusion Models for 3D Shape Generation12 Oct
  12. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep
2 more
  1. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan
  2. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
27 more
  1. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  2. 2018Demystifying MMD GANscited by 1,293
  3. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,205
  4. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  5. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 788
  6. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 785
  7. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 700
  8. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 689
  9. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  10. 2022Point-NeRF: Point-based Neural Radiance Fieldscited by 503
  11. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 488
  12. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  13. 2020Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstructioncited by 321
  14. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 306
  15. 2020Learning Gradient Fields for Shape Generationcited by 245
  16. 2022SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstructioncited by 216
  17. 2022From data to functa: Your data point is a function and you can treat it like onecited by 187
  18. 2021CodeNeRF: Disentangled Neural Radiance Fields for Object Categoriescited by 183
  19. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  20. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generationcited by 130
  21. 2022Diffusion-SDF: Text-to-Shape via Voxelized Diffusioncited by 128
  22. 2018Visual Object Networks: Image Generation with Disentangled 3D Representationcited by 124
  23. 2020SoftFlow: Probabilistic Framework for Normalizing Flow on Manifoldscited by 98
  24. 2020Discrete Point Flow Networks for Efficient Point Cloud Generationcited by 87
  25. 2023Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Renderingcited by 84
  26. 2022Disentangled3D: Learning a 3D Generative Model with Disentangled Geometry and Appearance from Monocular Imagescited by 39
  27. 2023SimNP: Learning Self-Similarity Priors Between Neural Pointscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 2; 2026: 1.

11 papers.

  1. 2024latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstructioncited by 118
  2. 2025MEt3R: Measuring Multi-View Consistency in Generated Imagescited by 95
  3. 2024InterTrack: Tracking Human Object Interaction without Object Templatescited by 23
  4. 2024GaussianAnything: Interactive Point Cloud Flow Matching For 3D Object Generationcited by 12
  5. 2024Spurfies: Sparse Surface Reconstruction using Local Geometry Priorscited by 8
  6. 2024Gaussian Scenes: Pose-Free Sparse-View Scene Reconstruction using Depth-Enhanced Diffusion Priorscited by 8
  7. 2024Sp2360: Sparse-view 360 Scene Reconstruction using Cascaded 2D Diffusion Priorscited by 7
  8. 2024Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulationcited by 6
  9. 2026Mix3R: Mixing Feed-forward Reconstruction and Generative 3D Priors for Joint Multi-view Aligned 3D Reconstruction and Pose Estimationcited by 4
  10. 2024SLayR: Scene Layout Generation with Rectified Flowcited by 2
  11. 2025InJecteD: Analyzing Trajectories and Drift Dynamics in Denoising Diffusion Probabilistic Models for 2D Point Cloud Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior
  2. 2024Efficient and Scalable Point Cloud Generation with Sparse Point-Voxel Diffusion Models
  3. 2023Breathing New Life into 3D Assets with Generative Repainting
  4. 2026Rethinking 3D Shape Generation: Diffusion over Superquadrics
  5. 2024Deformable 3D Shape Diffusion Model
  6. 2023DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation
  7. 2024Diff-PCC: Diffusion-based Neural Compression for 3D Point Clouds
  8. 2023Controllable Mesh Generation Through Sparse Latent Point Diffusion Models