# Generative 3D Part Assembly via Part-Whole-Hierarchy Message Passing

arXiv: https://arxiv.org/abs/2402.17464 · HTML: https://mlsanity.com/paper/2402.17464/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 27 Feb 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions](https://mlsanity.com/paper/1905.06292/) (2019)
- [PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes](https://mlsanity.com/paper/1911.10949/) (2019)
- [SDM-NET: Deep Generative Network for Structured Deformable Mesh](https://mlsanity.com/paper/1908.04520/) (2019)
- [Learning to Generate Realistic LiDAR Point Clouds](https://mlsanity.com/paper/2209.03954/) (2022)
- [UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation](https://mlsanity.com/paper/2311.01448/) (2023)
- [Generative 3D Part Assembly via Dynamic Graph Learning](https://mlsanity.com/paper/2006.07793/) (2020)
- [Learning 3D Part Assembly from a Single Image](https://mlsanity.com/paper/2003.09754/) (2020)
- [Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly](https://mlsanity.com/paper/1910.13675/) (2019)
- [Neural Shape Mating: Self-Supervised Object Assembly with Adversarial Shape Priors](https://mlsanity.com/paper/2205.14886/) (2022)
- [Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly](https://mlsanity.com/paper/1903.01066/) (2019)
- [RGL-NET: A Recurrent Graph Learning framework for Progressive Part Assembly](https://mlsanity.com/paper/2107.12859/) (2021)
- [PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions](https://mlsanity.com/paper/2003.08624/) (2020)
- [3D Part Assembly Generation with Instance Encoded Transformer](https://mlsanity.com/paper/2207.01779/) (2022)
- [Learning to Scaffold the Development of Robotic Manipulation Skills](https://mlsanity.com/paper/1911.00969/) (2019)
- [Score-PA: Score-based 3D Part Assembly](https://mlsanity.com/paper/2309.04220/) (2023)
- [Learning Pose Estimation for High-Precision Robotic Assembly Using Simulated Depth Images](https://mlsanity.com/paper/1809.10699/) (2018)
- [Unsupervised 3D Shape Reconstruction by Part Retrieval and Assembly](https://mlsanity.com/paper/2303.01999/) (2023)

## Built on by (most cited first, 19 of 19)

- [PuzzleFusion++: Auto-agglomerative 3D Fracture Assembly by Denoise and Verify](https://mlsanity.com/paper/2406.00259/) (2024)
- [Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Models](https://mlsanity.com/paper/2502.10090/) (2025)
- [Assembler: Scalable 3D Part Assembly via Anchor Point Diffusion](https://mlsanity.com/paper/2506.17074/) (2025)
- [A Survey on Computational Solutions for Reconstructing Complete Objects by Reassembling Their Fractured Parts](https://mlsanity.com/paper/2410.14770/) (2024)
- [Manual-PA: Learning 3D Part Assembly from Instruction Diagrams](https://mlsanity.com/paper/2411.18011/) (2024)
- [Multi-scale Latent Point Consistency Models for 3D Shape Generation](https://mlsanity.com/paper/2412.19413/) (2024)
- [Jigsaw++: Imagining Complete Shape Priors for Object Reassembly](https://mlsanity.com/paper/2410.11816/) (2024)
- [Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Models](https://mlsanity.com/paper/2510.16344/) (2025)
- [Geometry and Perception Guided Gaussians for Multiview-consistent 3D Generation from a Single Image](https://mlsanity.com/paper/2506.21152/) (2025)
- [HierOctFusion: Multi-scale Octree-based 3D Shape Generation via Part-Whole-Hierarchy Message Passing](https://mlsanity.com/paper/2508.11106/) (2025)
- [Behave Your Motion: Habit-preserved Cross-category Animal Motion Transfer](https://mlsanity.com/paper/2507.07394/) (2025)
- [From Part to Whole: 3D Generative World Model with an Adaptive Structural Hierarchy](https://mlsanity.com/paper/2603.21557/) (2026)
- [Leveraging Pretrained Diffusion Models for Zero-Shot Part Assembly](https://mlsanity.com/paper/2505.00426/) (2025)
- [View-Consistent 3D Scene Editing via Dual-Path Structural Correspondense and Semantic Continuity](https://mlsanity.com/paper/2604.17801/) (2026)
- [PART: Learning 3D Part Assembly and Retrieval with Transformers](https://mlsanity.com/paper/2609.19872/) (2026)
- [AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents](https://mlsanity.com/paper/2609.40353/) (2026)
- [ScaffoldM3C: A Multimodal Sequential Monte Carlo Framework for Generative Stable Construction Planning](https://mlsanity.com/paper/2610.00487/) (2026)
- [Organize Primitives into Semantic Parts: Reinforcement Reasoning for 3D Segmentation](https://mlsanity.com/paper/2610.04602/) (2026)
- [LLM2R: From Large Language Models to 3D Reassemblers](https://mlsanity.com/paper/2610.18604/) (2026)

## Abstract (arXiv)

Generative 3D part assembly involves understanding part relationships and predicting their 6-DoF poses for assembling a realistic 3D shape. Prior work often focus on the geometry of individual parts, neglecting part-whole hierarchies of objects. Leveraging two key observations: 1) super-part poses provide strong hints about part poses, and 2) predicting super-part poses is easier due to fewer superparts, we propose a part-whole-hierarchy message passing network for efficient 3D part assembly. We first introduce super-parts by grouping geometrically similar parts without any semantic labels. Then we employ a part-whole hierarchical encoder, wherein a super-part encoder predicts latent super-part poses based on input parts. Subsequently, we transform the point cloud using the latent poses, feeding it to the part encoder for aggregating super-part information and reasoning about part relationships to predict all part poses. In training, only ground-truth part poses are required. During inference, the predicted latent poses of super-parts enhance interpretability. Experimental results on the PartNet dataset show that our method achieves state-of-the-art performance in part and connectivity accuracy and enables an interpretable hierarchical part assembly. Code is available at https://github.com/pkudba/3DHPA.