# ShapeCoder: Discovering Abstractions for Visual Programs from Unstructured Primitives

arXiv: https://arxiv.org/abs/2305.05661 · HTML: https://mlsanity.com/paper/2305.05661/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 65 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [DeepCAD: A Deep Generative Network for Computer-Aided Design Models](https://mlsanity.com/paper/2105.09492/) (2021)
- [DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation](https://mlsanity.com/paper/2007.11301/) (2020)
- [SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebooks](https://mlsanity.com/paper/2207.04632/) (2022)
- [Im2Vec: Synthesizing Vector Graphics without Vector Supervision](https://mlsanity.com/paper/2102.02798/) (2021)
- [CSGNet: Neural Shape Parser for Constructive Solid Geometry](https://mlsanity.com/paper/1712.08290/) (2017)
- [Write, Execute, Assess: Program Synthesis with a REPL](https://mlsanity.com/paper/1906.04604/) (2019)
- [Computer-Aided Design as Language](https://mlsanity.com/paper/2105.02769/) (2021)
- [UCSG-Net -- Unsupervised Discovering of Constructive Solid Geometry Tree](https://mlsanity.com/paper/2006.09102/) (2020)
- [Learning to Infer Graphics Programs from Hand-Drawn Images](https://mlsanity.com/paper/1707.09627/) (2017)
- [Learning to Infer and Execute 3D Shape Programs](https://mlsanity.com/paper/1901.02875/) (2019)
- [SketchGen: Generating Constrained CAD Sketches](https://mlsanity.com/paper/2106.02711/) (2021)
- [Vitruvion: A Generative Model of Parametric CAD Sketches](https://mlsanity.com/paper/2109.14124/) (2021)
- [CAPRI-Net: Learning Compact CAD Shapes with Adaptive Primitive Assembly](https://mlsanity.com/paper/2104.05652/) (2021)
- [Top-Down Synthesis for Library Learning](https://mlsanity.com/paper/2211.16605/) (2022)
- [CSG-Stump: A Learning Friendly CSG-Like Representation for Interpretable Shape Parsing](https://mlsanity.com/paper/2108.11305/) (2021)
- [Inferring CAD Modeling Sequences Using Zone Graphs](https://mlsanity.com/paper/2104.03900/) (2021)
- [Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds](https://mlsanity.com/paper/2106.03437/) (2021)
- [ExtrudeNet: Unsupervised Inverse Sketch-and-Extrude for Shape Parsing](https://mlsanity.com/paper/2209.15632/) (2022)
- [Neurosymbolic Models for Computer Graphics](https://mlsanity.com/paper/2304.10320/) (2023)
- [An Inverse Procedural Modeling Pipeline for SVBRDF Maps](https://mlsanity.com/paper/2109.06395/) (2021)
- [babble: Learning Better Abstractions with E-Graphs and Anti-Unification](https://mlsanity.com/paper/2212.04596/) (2022)
- [A Multi-Implicit Neural Representation for Fonts](https://mlsanity.com/paper/2106.06866/) (2021)
- [PLAD: Learning to Infer Shape Programs with Pseudo-Labels and Approximate Distributions](https://mlsanity.com/paper/2011.13045/) (2020)
- [GeoCode: Interpretable Shape Programs](https://mlsanity.com/paper/2212.11715/) (2022)
- [Inverse Procedural Modeling of Facade Layouts](https://mlsanity.com/paper/1308.0419/) (2013)
- [Photo-to-Shape Material Transfer for Diverse Structures](https://mlsanity.com/paper/2205.04018/) (2022)

## Built on by (most cited first, 23 of 23)

- [Img2CAD: Reverse Engineering 3D CAD Models from Images through VLM-Assisted Conditional Factorization](https://mlsanity.com/paper/2408.01437/) (2024)
- [LL3M: Large Language 3D Modelers](https://mlsanity.com/paper/2508.08228/) (2025)
- [MeshCoder: LLM-Powered Structured Mesh Code Generation from Point Clouds](https://mlsanity.com/paper/2508.14879/) (2025)
- [Design2GarmentCode: Turning Design Concepts to Tangible Garments Through Program Synthesis](https://mlsanity.com/paper/2412.08603/) (2024)
- [CADTalk: An Algorithm and Benchmark for Semantic Commenting of CAD Programs](https://mlsanity.com/paper/2311.16703/) (2023)
- [3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code](https://mlsanity.com/paper/2606.01057/) (2026)
- [Text-to-3D Shape Generation](https://mlsanity.com/paper/2403.13289/) (2024)
- [DiffCSG: Differentiable CSG via Rasterization](https://mlsanity.com/paper/2409.01421/) (2024)
- [ParSEL: Parameterized Shape Editing with Language](https://mlsanity.com/paper/2405.20319/) (2024)
- [ArcPro: Architectural Programs for Structured 3D Abstraction of Sparse Points](https://mlsanity.com/paper/2503.02745/) (2025)
- [From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries](https://mlsanity.com/paper/2510.10292/) (2025)
- [ShapeLib: Designing a library of programmatic 3D shape abstractions with Large Language Models](https://mlsanity.com/paper/2502.08884/) (2025)
- [CADReview: Automatically Reviewing CAD Programs with Error Detection and Correction](https://mlsanity.com/paper/2505.22304/) (2025)
- [Gesturing Toward Abstraction: Multimodal Convention Formation in Collaborative Physical Tasks](https://mlsanity.com/paper/2602.08914/) (2026)
- [Mamba-CAD: State Space Model For 3D Computer-Aided Design Generative Modeling](https://mlsanity.com/paper/2603.00439/) (2026)
- [AbstractBeam: Enhancing Bottom-Up Program Synthesis using Library Learning](https://mlsanity.com/paper/2405.17514/) (2024)
- [StructRe: Rewriting for Structured Shape Modeling](https://mlsanity.com/paper/2311.17510/) (2023)
- [Parameterize Structure with Differentiable Template for 3D Shape Generation](https://mlsanity.com/paper/2410.10399/) (2024)
- [Synthesizing 3D Abstractions by Inverting Procedural Buildings with Transformers](https://mlsanity.com/paper/2501.17044/) (2025)
- [PLLM: Pseudo-Labeling Large Language Models for CAD Program Synthesis](https://mlsanity.com/paper/2602.12561/) (2026)
- [Pattern Analogies: Learning to Perform Programmatic Image Edits by Analogy](https://mlsanity.com/paper/2412.12463/) (2024)
- [FACT: Fidelity-Aware Construction of Articulated Twins](https://mlsanity.com/paper/2609.37067/) (2026)
- [Co-DSL: Human-LLM Co-design of Reusable Spatial Concepts for Layout Generation](https://mlsanity.com/paper/2610.15290/) (2026)

## Abstract (arXiv)

Programs are an increasingly popular representation for visual data, exposing compact, interpretable structure that supports manipulation. Visual programs are usually written in domain-specific languages (DSLs). Finding "good" programs, that only expose meaningful degrees of freedom, requires access to a DSL with a "good" library of functions, both of which are typically authored by domain experts. We present ShapeCoder, the first system capable of taking a dataset of shapes, represented with unstructured primitives, and jointly discovering (i) useful abstraction functions and (ii) programs that use these abstractions to explain the input shapes. The discovered abstractions capture common patterns (both structural and parametric) across the dataset, so that programs rewritten with these abstractions are more compact, and expose fewer degrees of freedom. ShapeCoder improves upon previous abstraction discovery methods, finding better abstractions, for more complex inputs, under less stringent input assumptions. This is principally made possible by two methodological advancements: (a) a shape to program recognition network that learns to solve sub-problems and (b) the use of e-graphs, augmented with a conditional rewrite scheme, to determine when abstractions with complex parametric expressions can be applied, in a tractable manner. We evaluate ShapeCoder on multiple datasets of 3D shapes, where primitive decompositions are either parsed from manual annotations or produced by an unsupervised cuboid abstraction method. In all domains, ShapeCoder discovers a library of abstractions that capture high-level relationships, remove extraneous degrees of freedom, and achieve better dataset compression compared with alternative approaches. Finally, we investigate how programs rewritten to use discovered abstractions prove useful for downstream tasks.