mlsanityData to 7 Oct 2026

LION: Latent Point Diffusion Models for 3D Shape Generation

arXiv 2210.06978 ↗First posted 12 Oct 2022cs.CV cs.LG stat.MLcited by 552

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Denoising diffusion models (DDMs) have shown promising results in 3D point cloud synthesis. To advance 3D DDMs and make them useful for digital artists, we require (i) high generation quality, (ii) flexibility for manipulation and applications such as conditional synthesis and shape interpolation, and (iii) the ability to output smooth surfaces or meshes. To this end, we introduce the hierarchical Latent Point Diffusion Model (LION) for 3D shape generation. LION is set up as a variational autoencoder (VAE) with a hierarchical latent space that combines a global shape latent representation with a point-structured latent space. For generation, we train two hierarchical DDMs in these latent spaces. The hierarchical VAE approach boosts performance compared to DDMs that operate on point clouds directly, while the point-structured latents are still ideally suited for DDM-based modeling. Experimentally, LION achieves state-of-the-art generation performance on multiple ShapeNet benchmarks. Furthermore, our VAE framework allows us to easily use LION for different relevant tasks: LION excels at multimodal shape denoising and voxel-conditioned synthesis, and it can be adapted for text- and image-driven 3D generation. We also demonstrate shape autoencoding and latent shape interpolation, and we augment LION with modern surface reconstruction techniques to generate smooth 3D meshes. We hope that LION provides a powerful tool for artists working with 3D shapes due to its high-quality generation, flexibility, and surface reconstruction. Project page and code: https://nv-tlabs.github.io/LION.

Key facts. Cited by 552 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 115 of its 147 references that are in the map. First posted on arXiv 12 Oct 2022.

Builds on

115 of 147 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion7 Dec
  2. 2021Diffusion Probabilistic Models for 3D Point Cloud Generation2 Mar

Other references · 113

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  12. 2017Squeeze-and-Excitation Networkscited by 4,763
38 more
  1. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  3. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  4. 2018Neural Ordinary Differential Equationscited by 3,733
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  6. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  7. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  8. 2015Variational Inference with Normalizing Flowscited by 2,161
  9. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  10. 2016Density estimation using Real NVPcited by 2,029
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  12. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  13. 2018Group Normalizationcited by 1,514
  14. 2020Point Transformercited by 1,462
  15. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  16. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  17. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  18. 2021Variational Diffusion Modelscited by 1,145
  19. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  20. 2019An Introduction to Variational Autoencoderscited by 997
  21. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  22. 2020Convolutional Occupancy Networkscited by 760
  23. 2020pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesiscited by 708
  24. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 695
  25. 2020GRAF: Generative Radiance Fields for 3D-Aware Image Synthesiscited by 688
  26. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  27. 2021Score-based Generative Modeling in Latent Spacecited by 634
  28. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  29. 2018FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Modelscited by 584
  30. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  31. 2018AtlasNet: A Papier-Mach'e Approach to Learning 3D Surface Generationcited by 539
  32. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  33. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 515
  34. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  35. 2019PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flowscited by 487
  36. 2021StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesiscited by 477
  37. 2021Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesiscited by 459
  38. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representationcited by 447

50 most cited of 113 All 113 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 19; 2023: 164; 2024: 176; 2025: 114; 2026: 79.

552 papers.

Sort:

By area:

  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  2. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  3. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  4. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  5. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
  6. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  7. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  8. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  9. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  10. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  11. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  12. 20233DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Modelscited by 449
38 more
  1. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  2. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  3. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  4. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  5. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusioncited by 309
  6. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
  7. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusioncited by 283
  8. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  9. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  10. 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 254
  11. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 240
  12. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  13. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 228
  14. 2023SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Secondscited by 227
  15. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  16. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 214
  17. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusioncited by 204
  18. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 198
  19. 2023Geometric Latent Diffusion Models for 3D Molecule Generationcited by 185
  20. 2023XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchiescited by 183
  21. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  22. 2023RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3Dcited by 168
  23. 2023DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesiscited by 159
  24. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Priorcited by 157
  25. 2025Native and Compact Structured Latents for 3D Generationcited by 157
  26. 2023DreamAvatar: Text-and-Shape Guided 3D Human Avatar Generation via Diffusion Modelscited by 154
  27. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  28. 2022Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Modelscited by 151
  29. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  30. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusioncited by 137
  31. 2022Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functionscited by 133
  32. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  33. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generationcited by 129
  34. 2022Diffusion-SDF: Text-to-Shape via Voxelized Diffusioncited by 128
  35. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  36. 2022Differentially Private Diffusion Modelscited by 118
  37. 2023Consistent View Synthesis with Pose-Guided Diffusion Modelscited by 117
  38. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistencycited by 117

50 most cited of 552 All 552 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LTM3D: Bridging Token Spaces for Conditional 3D Generation with Auto-Regressive Diffusion Framework
  2. 2023Single Mesh Diffusion Models with Field Latents for Texture Generation
  3. 2024Dora: Sampling and Benchmarking for 3D Shape Variational Auto-Encoders
  4. 2025Repurposing 2D Diffusion Models for 3D Shape Completion
  5. 2025Geometry-Preserving Encoder/Decoder in Latent Generative Models
  6. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation
  7. 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer
  8. 2025From One to More: Contextual Part Latents for 3D Generation