# The Neurally-Guided Shape Parser: Grammar-based Labeling of 3D Shape Regions with Approximate Inference

arXiv: https://arxiv.org/abs/2106.12026 · HTML: https://mlsanity.com/paper/2106.12026/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 22 Jun 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [Residual Gated Graph ConvNets](https://mlsanity.com/paper/1711.07553/) (2017)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [CvxNet: Learnable Convex Decomposition](https://mlsanity.com/paper/1909.05736/) (2019)
- [SDM-NET: Deep Generative Network for Structured Deformable Mesh](https://mlsanity.com/paper/1908.04520/) (2019)
- [Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks](https://mlsanity.com/paper/2103.10429/) (2021)
- [BAE-NET: Branched Autoencoder for Shape Co-Segmentation](https://mlsanity.com/paper/1903.11228/) (2019)
- [One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation](https://mlsanity.com/paper/2104.02246/) (2021)
- [Robust Watertight Manifold Surface Generation Method for ShapeNet Models](https://mlsanity.com/paper/1802.01698/) (2018)
- [Label-Efficient Learning on Point Clouds using Approximate Convex Decompositions](https://mlsanity.com/paper/2003.13834/) (2020)
- [Neural Star Domain as Primitive Representation](https://mlsanity.com/paper/2010.11248/) (2020)
- [Beyond Fixed Grid: Learning Geometric Image Representation with a Deformable Grid](https://mlsanity.com/paper/2008.09269/) (2020)
- [Learning Hierarchical Shape Segmentation and Labeling from Online Repositories](https://mlsanity.com/paper/1705.01661/) (2017)
- [Learning to Group and Label Fine-Grained Shape Components](https://mlsanity.com/paper/1809.05050/) (2018)
- [Hierarchy Denoising Recursive Autoencoders for 3D Scene Layout Prediction](https://mlsanity.com/paper/1903.03757/) (2019)
- [Compositionally Generalizable 3D Structure Prediction](https://mlsanity.com/paper/2012.02493/) (2020)
- [Learning Point Embeddings from Shape Repositories for Few-Shot Segmentation](https://mlsanity.com/paper/1910.01269/) (2019)
- [Superpixel-based Semantic Segmentation Trained by Statistical Process Control](https://mlsanity.com/paper/1706.10071/) (2017)

## Built on by (most cited first, 4 of 4)

- [HAL3D: Hierarchical Active Learning for Fine-Grained 3D Part Labeling](https://mlsanity.com/paper/2301.10460/) (2023)
- [Seg&Struct: The Interplay Between Part Segmentation and Structure Inference for 3D Shape Parsing](https://mlsanity.com/paper/2211.00382/) (2022)
- [SHRED: 3D Shape Region Decomposition with Learned Local Operations](https://mlsanity.com/paper/2206.03480/) (2022)
- [Material Magic Wand: Material-Aware Grouping of 3D Parts in Untextured Meshes](https://mlsanity.com/paper/2603.17370/) (2026)

## Abstract (arXiv)

We propose the Neurally-Guided Shape Parser (NGSP), a method that learns how to assign fine-grained semantic labels to regions of a 3D shape. NGSP solves this problem via MAP inference, modeling the posterior probability of a label assignment conditioned on an input shape with a learned likelihood function. To make this search tractable, NGSP employs a neural guide network that learns to approximate the posterior. NGSP finds high-probability label assignments by first sampling proposals with the guide network and then evaluating each proposal under the full likelihood. We evaluate NGSP on the task of fine-grained semantic segmentation of manufactured 3D shapes from PartNet, where shapes have been decomposed into regions that correspond to part instance over-segmentations. We find that NGSP delivers significant performance improvements over comparison methods that (i) use regions to group per-point predictions, (ii) use regions as a self-supervisory signal or (iii) assign labels to regions under alternative formulations. Further, we show that NGSP maintains strong performance even with limited labeled data or noisy input shape regions. Finally, we demonstrate that NGSP can be directly applied to CAD shapes found in online repositories and validate its effectiveness with a perceptual study.