# ShapeFusion: A 3D diffusion model for localized shape editing

arXiv: https://arxiv.org/abs/2403.19773 · HTML: https://mlsanity.com/paper/2403.19773/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 34 of its 53 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Diffusion Probabilistic Models for 3D Point Cloud Generation](https://mlsanity.com/paper/2103.01458/) (2021)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [3D Morphable Face Models -- Past, Present and Future](https://mlsanity.com/paper/1909.01815/) (2019)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [MeshDiffusion: Score-based Generative 3D Mesh Modeling](https://mlsanity.com/paper/2303.08133/) (2023)
- [STAR: Sparse Trained Articulated Human Body Regressor](https://mlsanity.com/paper/2008.08535/) (2020)
- [Shape As Points: A Differentiable Poisson Solver](https://mlsanity.com/paper/2106.03452/) (2021)
- [A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation](https://mlsanity.com/paper/2104.07645/) (2021)
- [Disentangled Person Image Generation](https://mlsanity.com/paper/1712.02621/) (2017)
- [Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation](https://mlsanity.com/paper/1905.02876/) (2019)
- [Disentangling factors of variation in deep representations using adversarial training](https://mlsanity.com/paper/1611.03383/) (2016)
- [Learning Neural Parametric Head Models](https://mlsanity.com/paper/2212.02761/) (2022)
- [Towards a complete 3D morphable model of the human head](https://mlsanity.com/paper/1911.08008/) (2019)
- [Unsupervised Shape and Pose Disentanglement for 3D Meshes](https://mlsanity.com/paper/2007.11341/) (2020)
- [Combining 3D Morphable Models: A Large scale Face-and-Head Model](https://mlsanity.com/paper/1903.03785/) (2019)
- [Neural Sign Actors: A diffusion model for 3D sign language production from text](https://mlsanity.com/paper/2312.02702/) (2023)
- [Neural Template: Topology-aware Reconstruction and Disentangled Generation of 3D Meshes](https://mlsanity.com/paper/2206.04942/) (2022)
- [Neural Face Rigging for Animating and Retargeting Facial Meshes in the Wild](https://mlsanity.com/paper/2305.08296/) (2023)
- [Disentangled Human Body Embedding Based on Deep Hierarchical Neural Network](https://mlsanity.com/paper/1905.05622/) (2019)
- [Intrinsic-Extrinsic Preserved GANs for Unsupervised 3D Pose Transfer](https://mlsanity.com/paper/2108.07520/) (2021)
- [3D Shape Variational Autoencoder Latent Disentanglement via Mini-Batch Feature Swapping for Bodies and Faces](https://mlsanity.com/paper/2111.12448/) (2021)
- [ILSH: The Imperial Light-Stage Head Dataset for Human Head View Synthesis](https://mlsanity.com/paper/2310.03952/) (2023)
- [Learning to Generate Customized Dynamic 3D Facial Expressions](https://mlsanity.com/paper/2007.09805/) (2020)

## Built on by (most cited first, 7 of 7)

- [3D Mesh Editing using Masked LRMs](https://mlsanity.com/paper/2412.08641/) (2024)
- [Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance](https://mlsanity.com/paper/2501.05379/) (2025)
- [ImHead: A Large-scale Implicit Morphable Model for Localized Head Modeling](https://mlsanity.com/paper/2510.10793/) (2025)
- [BFSM: 3D Bidirectional Face-Skull Morphable Model](https://mlsanity.com/paper/2509.24577/) (2025)
- [Multimodal Conditional 3D Face Geometry Generation](https://mlsanity.com/paper/2407.01074/) (2024)
- [CompoSE: Compositional Synthesis and Editing of 3D Shapes via Part-Aware Control](https://mlsanity.com/paper/2605.19350/) (2026)
- [ManifoldSplat: Language-Guided Semantic Shape Editing of 3D Gaussian Head Avatars](https://mlsanity.com/paper/2610.03599/) (2026)

## Abstract (arXiv)

In the realm of 3D computer vision, parametric models have emerged as a ground-breaking methodology for the creation of realistic and expressive 3D avatars. Traditionally, they rely on Principal Component Analysis (PCA), given its ability to decompose data to an orthonormal space that maximally captures shape variations. However, due to the orthogonality constraints and the global nature of PCA's decomposition, these models struggle to perform localized and disentangled editing of 3D shapes, which severely affects their use in applications requiring fine control such as face sculpting. In this paper, we leverage diffusion models to enable diverse and fully localized edits on 3D meshes, while completely preserving the un-edited regions. We propose an effective diffusion masking training strategy that, by design, facilitates localized manipulation of any shape region, without being limited to predefined regions or to sparse sets of predefined control vertices. Following our framework, a user can explicitly set their manipulation region of choice and define an arbitrary set of vertices as handles to edit a 3D mesh. Compared to the current state-of-the-art our method leads to more interpretable shape manipulations than methods relying on latent code state, greater localization and generation diversity while offering faster inference than optimization based approaches. Project page: https://rolpotamias.github.io/Shapefusion/