# Seg&Struct: The Interplay Between Part Segmentation and Structure Inference for 3D Shape Parsing

arXiv: https://arxiv.org/abs/2211.00382 · HTML: https://mlsanity.com/paper/2211.00382/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 70 references that are in the map.
- First posted on arXiv 1 Nov 2022.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation](https://mlsanity.com/paper/2004.01658/) (2020)
- [BSP-Net: Generating Compact Meshes via Binary Space Partitioning](https://mlsanity.com/paper/1911.06971/) (2019)
- [SoftGroup for 3D Instance Segmentation on Point Clouds](https://mlsanity.com/paper/2203.01509/) (2022)
- [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021)
- [OccuSeg: Occupancy-aware 3D Instance Segmentation](https://mlsanity.com/paper/2003.06537/) (2020)
- [SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation](https://mlsanity.com/paper/1711.08588/) (2017)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [SDM-NET: Deep Generative Network for Structured Deformable Mesh](https://mlsanity.com/paper/1908.04520/) (2019)
- [Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images](https://mlsanity.com/paper/1511.02300/) (2015)
- [Associatively Segmenting Instances and Semantics in Point Clouds](https://mlsanity.com/paper/1902.09852/) (2019)
- [IM2CAD](https://mlsanity.com/paper/1608.05137/) (2016)
- [Learning Unsupervised Hierarchical Part Decomposition of 3D Objects from a Single RGB Image](https://mlsanity.com/paper/2004.01176/) (2020)
- [SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation](https://mlsanity.com/paper/1612.00606/) (2016)
- [Learning to Infer and Execute 3D Shape Programs](https://mlsanity.com/paper/1901.02875/) (2019)
- [3D Shape Segmentation with Projective Convolutional Networks](https://mlsanity.com/paper/1612.02808/) (2016)
- [StructureNet: Hierarchical Graph Networks for 3D Shape Generation](https://mlsanity.com/paper/1908.00575/) (2019)
- [Im2Struct: Recovering 3D Shape Structure from a Single RGB Image](https://mlsanity.com/paper/1804.05469/) (2018)
- [Point Cloud Instance Segmentation using Probabilistic Embeddings](https://mlsanity.com/paper/1912.00145/) (2019)
- [MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization](https://mlsanity.com/paper/2101.06605/) (2021)
- [PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation](https://mlsanity.com/paper/1903.00709/) (2019)
- [Joint Learning of 3D Shape Retrieval and Deformation](https://mlsanity.com/paper/2101.07889/) (2021)
- [DSG-Net: Learning Disentangled Structure and Geometry for 3D Shape Generation](https://mlsanity.com/paper/2008.05440/) (2020)
- [Learning to Group: A Bottom-Up Framework for 3D Part Discovery in Unseen Categories](https://mlsanity.com/paper/2002.06478/) (2020)
- [SCORES: Shape Composition with Recursive Substructure Priors](https://mlsanity.com/paper/1809.05398/) (2018)
- [StructEdit: Learning Structural Shape Variations](https://mlsanity.com/paper/1911.11098/) (2019)

## Built on by (most cited first, 2 of 2)

- [StructRe: Rewriting for Structured Shape Modeling](https://mlsanity.com/paper/2311.17510/) (2023)
- [BoxSplitGen: A Generative Model for 3D Part Bounding Boxes in Varying Granularity](https://mlsanity.com/paper/2602.20666/) (2026)

## Abstract (arXiv)

We propose Seg&Struct, a supervised learning framework leveraging the interplay between part segmentation and structure inference and demonstrating their synergy in an integrated framework. Both part segmentation and structure inference have been extensively studied in the recent deep learning literature, while the supervisions used for each task have not been fully exploited to assist the other task. Namely, structure inference has been typically conducted with an autoencoder that does not leverage the point-to-part associations. Also, segmentation has been mostly performed without structural priors that tell the plausibility of the output segments. We present how these two tasks can be best combined while fully utilizing supervision to improve performance. Our framework first decomposes a raw input shape into part segments using an off-the-shelf algorithm, whose outputs are then mapped to nodes in a part hierarchy, establishing point-to-part associations. Following this, ours predicts the structural information, e.g., part bounding boxes and part relationships. Lastly, the segmentation is rectified by examining the confusion of part boundaries using the structure-based part features. Our experimental results based on the StructureNet and PartNet demonstrate that the interplay between the two tasks results in remarkable improvements in both tasks: 27.91% in structure inference and 0.5% in segmentation.