# Controllable Mesh Generation Through Sparse Latent Point Diffusion Models

arXiv: https://arxiv.org/abs/2303.07938 · HTML: https://mlsanity.com/paper/2303.07938/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 38 of its 45 references that are in the map.
- First posted on arXiv 14 Mar 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 38)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion](https://mlsanity.com/paper/2003.01456/) (2020)
- [Learning Gradient Fields for Shape Generation](https://mlsanity.com/paper/2008.06520/) (2020)
- [AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation](https://mlsanity.com/paper/2203.09516/) (2022)
- [Shape As Points: A Differentiable Poisson Solver](https://mlsanity.com/paper/2106.03452/) (2021)
- [3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions](https://mlsanity.com/paper/1905.06292/) (2019)
- [Deep Geometric Prior for Surface Reconstruction](https://mlsanity.com/paper/1811.10943/) (2018)
- [Neural Cages for Detail-Preserving 3D Deformations](https://mlsanity.com/paper/1912.06395/) (2019)
- [Pixel2Mesh++: Multi-View 3D Mesh Generation via Deformation](https://mlsanity.com/paper/1908.01491/) (2019)
- [A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion](https://mlsanity.com/paper/2112.03530/) (2021)
- [SoftFlow: Probabilistic Framework for Normalizing Flow on Manifolds](https://mlsanity.com/paper/2006.04604/) (2020)
- [SDF-StyleGAN: Implicit SDF-Based StyleGAN for 3D Shape Generation](https://mlsanity.com/paper/2206.12055/) (2022)
- [Discrete Point Flow Networks for Efficient Point Cloud Generation](https://mlsanity.com/paper/2007.10170/) (2020)
- [Learning Deformable Tetrahedral Meshes for 3D Reconstruction](https://mlsanity.com/paper/2011.01437/) (2020)
- [Point Cloud GAN](https://mlsanity.com/paper/1810.05795/) (2018)
- [Neural Dual Contouring](https://mlsanity.com/paper/2202.01999/) (2022)
- [Neural Mesh Flow: 3D Manifold Mesh Generation via Diffeomorphic Flows](https://mlsanity.com/paper/2007.10973/) (2020)

## Built on by (most cited first, 30 of 41)

- [MeshAnything V2: Artist-Created Mesh Generation With Adjacent Mesh Tokenization](https://mlsanity.com/paper/2408.02555/) (2024)
- [MeshXL: Neural Coordinate Field for Generative 3D Foundation Models](https://mlsanity.com/paper/2405.20853/) (2024)
- [D4M: Dataset Distillation via Disentangled Diffusion Model](https://mlsanity.com/paper/2407.15138/) (2024)
- [Scaling Mesh Generation via Compressive Tokenization](https://mlsanity.com/paper/2411.07025/) (2024)
- [Towards Realistic Scene Generation with LiDAR Diffusion Models](https://mlsanity.com/paper/2404.00815/) (2024)
- [You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scale](https://mlsanity.com/paper/2412.06699/) (2024)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [PivotMesh: Generic 3D Mesh Generation via Pivot Vertices Guidance](https://mlsanity.com/paper/2405.16890/) (2024)
- [UGG: Unified Generative Grasping](https://mlsanity.com/paper/2311.16917/) (2023)
- [AI-Generated Content (AIGC) for Various Data Modalities: A Survey](https://mlsanity.com/paper/2308.14177/) (2023)
- [InstructP2P: Learning to Edit 3D Point Clouds with Text Instructions](https://mlsanity.com/paper/2306.07154/) (2023)
- [VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representation](https://mlsanity.com/paper/2307.16605/) (2023)
- [Topology-Aware Latent Diffusion for 3D Shape Generation](https://mlsanity.com/paper/2401.17603/) (2024)
- [GaussianAnything: Interactive Point Cloud Flow Matching For 3D Object Generation](https://mlsanity.com/paper/2411.08033/) (2024)
- [Surf-D: Generating High-Quality Surfaces of Arbitrary Topologies Using Diffusion Models](https://mlsanity.com/paper/2311.17050/) (2023)
- [CNS-Edit: 3D Shape Editing via Coupled Neural Shape Optimization](https://mlsanity.com/paper/2402.02313/) (2024)
- [GetMesh: A Controllable Model for High-quality Mesh Generation and Manipulation](https://mlsanity.com/paper/2403.11990/) (2024)
- [Linear combinations of latents in generative models: subspaces and beyond](https://mlsanity.com/paper/2408.08558/) (2024)
- [FastGrasp: Efficient Grasp Synthesis with Diffusion](https://mlsanity.com/paper/2411.14786/) (2024)
- [DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creation](https://mlsanity.com/paper/2412.15200/) (2024)
- [AnimateMe: 4D Facial Expressions via Diffusion Models](https://mlsanity.com/paper/2403.17213/) (2024)
- [Jigsaw++: Imagining Complete Shape Priors for Object Reassembly](https://mlsanity.com/paper/2410.11816/) (2024)
- [Sharp-It: A Multi-view to Multi-view Diffusion Model for 3D Synthesis and Manipulation](https://mlsanity.com/paper/2412.02631/) (2024)
- [Self-Supervised Multi-Frame Neural Scene Flow](https://mlsanity.com/paper/2403.16116/) (2024)
- [DiffSurf: A Transformer-based Diffusion Model for Generating and Reconstructing 3D Surfaces in Pose](https://mlsanity.com/paper/2408.14860/) (2024)
- [GenUDC: High Quality 3D Mesh Generation with Unsigned Dual Contouring Representation](https://mlsanity.com/paper/2410.17802/) (2024)
- [JADE: Joint-aware Latent Diffusion for 3D Human Generative Modeling](https://mlsanity.com/paper/2412.20470/) (2024)
- [MergeNet: Explicit Mesh Reconstruction from Sparse Point Clouds via Edge Prediction](https://mlsanity.com/paper/2407.11610/) (2024)
- [DiffCom: Decoupled Sparse Priors Guided Diffusion Compression for Point Clouds](https://mlsanity.com/paper/2411.13860/) (2024)
- [PaNDaS: Learnable Deformation Modeling with Localized Control](https://mlsanity.com/paper/2412.02306/) (2024)

## Abstract (arXiv)

Mesh generation is of great value in various applications involving computer graphics and virtual content, yet designing generative models for meshes is challenging due to their irregular data structure and inconsistent topology of meshes in the same category. In this work, we design a novel sparse latent point diffusion model for mesh generation. Our key insight is to regard point clouds as an intermediate representation of meshes, and model the distribution of point clouds instead. While meshes can be generated from point clouds via techniques like Shape as Points (SAP), the challenges of directly generating meshes can be effectively avoided. To boost the efficiency and controllability of our mesh generation method, we propose to further encode point clouds to a set of sparse latent points with point-wise semantic meaningful features, where two DDPMs are trained in the space of sparse latent points to respectively model the distribution of the latent point positions and features at these latent points. We find that sampling in this latent space is faster than directly sampling dense point clouds. Moreover, the sparse latent points also enable us to explicitly control both the overall structures and local details of the generated meshes. Extensive experiments are conducted on the ShapeNet dataset, where our proposed sparse latent point diffusion model achieves superior performance in terms of generation quality and controllability when compared to existing methods.