# Learning to Infer and Execute 3D Shape Programs

arXiv: https://arxiv.org/abs/1901.02875 · HTML: https://mlsanity.com/paper/1901.02875/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2019.

## Built on by (most cited first, 30 of 68)

- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [DeepCAD: A Deep Generative Network for Computer-Aided Design Models](https://mlsanity.com/paper/2105.09492/) (2021)
- [SkexGen: Autoregressive Generation of CAD Construction Sequences with Disentangled Codebooks](https://mlsanity.com/paper/2207.04632/) (2022)
- [Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks](https://mlsanity.com/paper/2103.10429/) (2021)
- [Hierarchical Neural Coding for Controllable CAD Model Generation](https://mlsanity.com/paper/2307.00149/) (2023)
- [CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM](https://mlsanity.com/paper/2411.04954/) (2024)
- [SolidGen: An Autoregressive Model for Direct B-rep Synthesis](https://mlsanity.com/paper/2203.13944/) (2022)
- [Reliable, Adaptable, and Attributable Language Models with Retrieval](https://mlsanity.com/paper/2403.03187/) (2024)
- [SketchGen: Generating Constrained CAD Sketches](https://mlsanity.com/paper/2106.02711/) (2021)
- [CAPRI-Net: Learning Compact CAD Shapes with Adaptive Primitive Assembly](https://mlsanity.com/paper/2104.05652/) (2021)
- [Point2Cyl: Reverse Engineering 3D Objects from Point Clouds to Extrusion Cylinders](https://mlsanity.com/paper/2112.09329/) (2021)
- [Engineering Sketch Generation for Computer-Aided Design](https://mlsanity.com/paper/2104.09621/) (2021)
- [Reasoning Like Program Executors](https://mlsanity.com/paper/2201.11473/) (2022)
- [Towards High-Fidelity Single-view Holistic Reconstruction of Indoor Scenes](https://mlsanity.com/paper/2207.08656/) (2022)
- [Reconstructing editable prismatic CAD from rounded voxel models](https://mlsanity.com/paper/2209.01161/) (2022)
- [D2CSG: Unsupervised Learning of Compact CSG Trees with Dual Complements and Dropouts](https://mlsanity.com/paper/2301.11497/) (2023)
- [The Scene Language: Representing Scenes with Programs, Words, and Embeddings](https://mlsanity.com/paper/2410.16770/) (2024)
- [Latent Execution for Neural Program Synthesis](https://mlsanity.com/paper/2107.00101/) (2021)
- [Can Large Language Models Understand Symbolic Graphics Programs?](https://mlsanity.com/paper/2408.08313/) (2024)
- [HAKE: A Knowledge Engine Foundation for Human Activity Understanding](https://mlsanity.com/paper/2202.06851/) (2022)
- [Outline, Then Details: Syntactically Guided Coarse-To-Fine Code Generation](https://mlsanity.com/paper/2305.00909/) (2023)
- [TikZero: Zero-Shot Text-Guided Graphics Program Synthesis](https://mlsanity.com/paper/2503.11509/) (2025)
- [ShapeCoder: Discovering Abstractions for Visual Programs from Unstructured Primitives](https://mlsanity.com/paper/2305.05661/) (2023)
- [ProTo: Program-Guided Transformer for Program-Guided Tasks](https://mlsanity.com/paper/2110.00804/) (2021)
- [DTGBrepGen: A Novel B-rep Generative Model through Decoupling Topology and Geometry](https://mlsanity.com/paper/2503.13110/) (2025)
- [Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs](https://mlsanity.com/paper/2301.12950/) (2023)
- [IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Rendering](https://mlsanity.com/paper/2506.23329/) (2025)
- [MeshCoder: LLM-Powered Structured Mesh Code Generation from Point Clouds](https://mlsanity.com/paper/2508.14879/) (2025)
- [Design2GarmentCode: Turning Design Concepts to Tangible Garments Through Program Synthesis](https://mlsanity.com/paper/2412.08603/) (2024)
- [ANISE: Assembly-based Neural Implicit Surface rEconstruction](https://mlsanity.com/paper/2205.13682/) (2022)

## Abstract (arXiv)

Human perception of 3D shapes goes beyond reconstructing them as a set of points or a composition of geometric primitives: we also effortlessly understand higher-level shape structure such as the repetition and reflective symmetry of object parts. In contrast, recent advances in 3D shape sensing focus more on low-level geometry but less on these higher-level relationships. In this paper, we propose 3D shape programs, integrating bottom-up recognition systems with top-down, symbolic program structure to capture both low-level geometry and high-level structural priors for 3D shapes. Because there are no annotations of shape programs for real shapes, we develop neural modules that not only learn to infer 3D shape programs from raw, unannotated shapes, but also to execute these programs for shape reconstruction. After initial bootstrapping, our end-to-end differentiable model learns 3D shape programs by reconstructing shapes in a self-supervised manner. Experiments demonstrate that our model accurately infers and executes 3D shape programs for highly complex shapes from various categories. It can also be integrated with an image-to-shape module to infer 3D shape programs directly from an RGB image, leading to 3D shape reconstructions that are both more accurate and more physically plausible.