# PyTorchGeoNodes: Enabling Differentiable Shape Programs for 3D Shape Reconstruction

arXiv: https://arxiv.org/abs/2404.10620 · HTML: https://mlsanity.com/paper/2404.10620/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [Splatter Image: Ultra-Fast Single-View 3D Reconstruction](https://mlsanity.com/paper/2312.13150/) (2023)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [Depth-Regularized Optimization for 3D Gaussian Splatting in Few-Shot Images](https://mlsanity.com/paper/2311.13398/) (2023)
- [DN-Splatter: Depth and Normal Priors for Gaussian Splatting and Meshing](https://mlsanity.com/paper/2403.17822/) (2024)
- [Recent Advances in 3D Gaussian Splatting](https://mlsanity.com/paper/2403.11134/) (2024)
- [IM2CAD](https://mlsanity.com/paper/1608.05137/) (2016)
- [Learning to Infer and Execute 3D Shape Programs](https://mlsanity.com/paper/1901.02875/) (2019)
- [Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve](https://mlsanity.com/paper/2007.13034/) (2020)
- [SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields](https://mlsanity.com/paper/2212.02501/) (2022)
- [Neurosymbolic Models for Computer Graphics](https://mlsanity.com/paper/2304.10320/) (2023)
- [Pixels, voxels, and views: A study of shape representations for single view 3D object shape prediction](https://mlsanity.com/paper/1804.06032/) (2018)
- [MonteFloor: Extending MCTS for Reconstructing Accurate Large-Scale Floor Plans](https://mlsanity.com/paper/2103.11161/) (2021)
- [GeoCode: Interpretable Shape Programs](https://mlsanity.com/paper/2212.11715/) (2022)
- [Understanding deep features with computer-generated imagery](https://mlsanity.com/paper/1506.01151/) (2015)
- [Automatically Annotating Indoor Images with CAD Models via RGB-D Scans](https://mlsanity.com/paper/2212.11796/) (2022)
- [Deep Exemplar 2D-3D Detection by Adapting from Real to Rendered Views](https://mlsanity.com/paper/1512.02497/) (2015)
- [A Coarse-to-Fine Model for 3D Pose Estimation and Sub-category Recognition](https://mlsanity.com/paper/1504.02764/) (2015)
- [MCTS with Refinement for Proposals Selection Games in Scene Understanding](https://mlsanity.com/paper/2207.03204/) (2022)
- [HOC-Search: Efficient CAD Model and Pose Retrieval from RGB-D Scans](https://mlsanity.com/paper/2309.06107/) (2023)

## Built on by (most cited first, 5 of 5)

- [LL3M: Large Language 3D Modelers](https://mlsanity.com/paper/2508.08228/) (2025)
- [DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation for High-quality 3D Asset Creation](https://mlsanity.com/paper/2412.15200/) (2024)
- [Can AI Understand the Language of Origami?](https://mlsanity.com/paper/2603.13856/) (2026)
- [GuideFlow3D: Optimization-Guided Rectified Flow For Appearance Transfer](https://mlsanity.com/paper/2510.16136/) (2025)
- [Proc3D: Procedural 3D Generation and Parametric Editing of 3D Shapes with Large Language Models](https://mlsanity.com/paper/2601.12234/) (2026)

## Abstract (arXiv)

We propose PyTorchGeoNodes, a differentiable module for reconstructing 3D objects and their parameters from images using interpretable shape programs. Unlike traditional CAD model retrieval, shape programs allow reasoning about semantic parameters, editing, and a low memory footprint. Despite their potential, shape programs for 3D scene understanding have been largely overlooked. Our key contribution is enabling gradient-based optimization by parsing shape programs, or more precisely procedural models designed in Blender, into efficient PyTorch code. While there are many possible applications of our PyTochGeoNodes, we show that a combination of PyTorchGeoNodes with genetic algorithm is a method of choice to optimize both discrete and continuous shape program parameters for 3D reconstruction and understanding of 3D object parameters. Our modular framework can be further integrated with other reconstruction algorithms, and we demonstrate one such integration to enable procedural Gaussian splatting. Our experiments on the ScanNet dataset show that our method achieves accurate reconstructions while enabling, until now, unseen level of 3D scene understanding.