# ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis

arXiv: https://arxiv.org/abs/2009.08026 · HTML: https://mlsanity.com/paper/2009.08026/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 19 of 19)

- [JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints](https://mlsanity.com/paper/2111.12772/) (2021)
- [3D and 4D World Modeling: A Survey](https://mlsanity.com/paper/2509.07996/) (2025)
- [RGL-NET: A Recurrent Graph Learning framework for Progressive Part Assembly](https://mlsanity.com/paper/2107.12859/) (2021)
- [Img2CAD: Reverse Engineering 3D CAD Models from Images through VLM-Assisted Conditional Factorization](https://mlsanity.com/paper/2408.01437/) (2024)
- [GaussianArt: Unified Modeling of Geometry and Motion for Articulated Objects](https://mlsanity.com/paper/2508.14891/) (2025)
- [CasaGPT: Cuboid Arrangement and Scene Assembly for Interior Design](https://mlsanity.com/paper/2504.19478/) (2025)
- [AutoMate: A Dataset and Learning Approach for Automatic Mating of CAD Assemblies](https://mlsanity.com/paper/2105.12238/) (2021)
- [SPAFormer: Sequential 3D Part Assembly with Transformers](https://mlsanity.com/paper/2403.05874/) (2024)
- [ArtiLatent: Realistic Articulated 3D Object Generation via Structured Latents](https://mlsanity.com/paper/2510.21432/) (2025)
- [Neural Shape Compiler: A Unified Framework for Transforming between Text, Point Cloud, and Program](https://mlsanity.com/paper/2212.12952/) (2022)
- [Explorable Mesh Deformation Subspaces from Unstructured Generative Models](https://mlsanity.com/paper/2310.07814/) (2023)
- [ArcPro: Architectural Programs for Structured 3D Abstraction of Sparse Points](https://mlsanity.com/paper/2503.02745/) (2025)
- [Learning Body-Aware 3D Shape Generative Models](https://mlsanity.com/paper/2112.07022/) (2021)
- [FaccAID: A Transformer Model for Neuro-Symbolic Facade Reconstruction](https://mlsanity.com/paper/2406.01829/) (2024)
- [A Survey of Methods for Converting Unstructured Data to CSG Models](https://mlsanity.com/paper/2305.01220/) (2023)
- [Seg&Struct: The Interplay Between Part Segmentation and Structure Inference for 3D Shape Parsing](https://mlsanity.com/paper/2211.00382/) (2022)
- [GaussianCAD: Robust Self-Supervised CAD Reconstruction from Three Orthographic Views Using 3D Gaussian Splatting](https://mlsanity.com/paper/2503.05161/) (2025)
- [Fabrication-Aware Reverse Engineering for Carpentry](https://mlsanity.com/paper/2107.09965/) (2021)
- [Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the Art](https://mlsanity.com/paper/2311.10129/) (2023)

## Abstract (arXiv)

Manually authoring 3D shapes is difficult and time consuming; generative models of 3D shapes offer compelling alternatives. Procedural representations are one such possibility: they offer high-quality and editable results but are difficult to author and often produce outputs with limited diversity. On the other extreme are deep generative models: given enough data, they can learn to generate any class of shape but their outputs have artifacts and the representation is not editable. In this paper, we take a step towards achieving the best of both worlds for novel 3D shape synthesis. We propose ShapeAssembly, a domain-specific "assembly-language" for 3D shape structures. ShapeAssembly programs construct shapes by declaring cuboid part proxies and attaching them to one another, in a hierarchical and symmetrical fashion. Its functions are parameterized with free variables, so that one program structure is able to capture a family of related shapes. We show how to extract ShapeAssembly programs from existing shape structures in the PartNet dataset. Then we train a deep generative model, a hierarchical sequence VAE, that learns to write novel ShapeAssembly programs. The program captures the subset of variability that is interpretable and editable. The deep model captures correlations across shape collections that are hard to express procedurally. We evaluate our approach by comparing shapes output by our generated programs to those from other recent shape structure synthesis models. We find that our generated shapes are more plausible and physically-valid than those of other methods. Additionally, we assess the latent spaces of these models, and find that ours is better structured and produces smoother interpolations. As an application, we use our generative model and differentiable program interpreter to infer and fit shape programs to unstructured geometry, such as point clouds.