# 3D Part Assembly Generation with Instance Encoded Transformer

arXiv: https://arxiv.org/abs/2207.01779 · HTML: https://mlsanity.com/paper/2207.01779/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 5 Jul 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [Attention Augmented Convolutional Networks](https://mlsanity.com/paper/1904.09925/) (2019)
- [PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes](https://mlsanity.com/paper/1911.10949/) (2019)
- [SDM-NET: Deep Generative Network for Structured Deformable Mesh](https://mlsanity.com/paper/1908.04520/) (2019)
- [Generative 3D Part Assembly via Dynamic Graph Learning](https://mlsanity.com/paper/2006.07793/) (2020)
- [Learning 3D Part Assembly from a Single Image](https://mlsanity.com/paper/2003.09754/) (2020)
- [Learning Part Generation and Assembly for Structure-aware Shape Synthesis](https://mlsanity.com/paper/1906.06693/) (2019)
- [RGL-NET: A Recurrent Graph Learning framework for Progressive Part Assembly](https://mlsanity.com/paper/2107.12859/) (2021)
- [Composite Shape Modeling via Latent Space Factorization](https://mlsanity.com/paper/1901.02968/) (2019)
- [GRASS: Generative Recursive Autoencoders for Shape Structures](https://mlsanity.com/paper/1705.02090/) (2017)
- [StructEdit: Learning Structural Shape Variations](https://mlsanity.com/paper/1911.11098/) (2019)
- [SAGNet:Structure-aware Generative Network for 3D-Shape Modeling](https://mlsanity.com/paper/1808.03981/) (2018)
- [PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions](https://mlsanity.com/paper/2003.08624/) (2020)
- [ComplementMe: Weakly-Supervised Component Suggestions for 3D Modeling](https://mlsanity.com/paper/1708.01841/) (2017)
- [Learning to Scaffold the Development of Robotic Manipulation Skills](https://mlsanity.com/paper/1911.00969/) (2019)
- [Learning Pose Estimation for High-Precision Robotic Assembly Using Simulated Depth Images](https://mlsanity.com/paper/1809.10699/) (2018)
- [Interpreting Contact Interactions to Overcome Failure in Robot Assembly Tasks](https://mlsanity.com/paper/2101.02725/) (2021)

## Built on by (most cited first, 19 of 19)

- [Going Denser with Open-Vocabulary Part Segmentation](https://mlsanity.com/paper/2305.11173/) (2023)
- [Generative 3D Part Assembly via Part-Whole-Hierarchy Message Passing](https://mlsanity.com/paper/2402.17464/) (2024)
- [PuzzleFusion++: Auto-agglomerative 3D Fracture Assembly by Denoise and Verify](https://mlsanity.com/paper/2406.00259/) (2024)
- [Scalable Geometric Fracture Assembly via Co-creation Space among Assemblers](https://mlsanity.com/paper/2312.12340/) (2023)
- [Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulation](https://mlsanity.com/paper/2504.06961/) (2025)
- [SPAFormer: Sequential 3D Part Assembly with Transformers](https://mlsanity.com/paper/2403.05874/) (2024)
- [IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos](https://mlsanity.com/paper/2411.11409/) (2024)
- [Planning Assembly Sequence with Graph Transformer](https://mlsanity.com/paper/2210.05236/) (2022)
- [A Survey on Computational Solutions for Reconstructing Complete Objects by Reassembling Their Fractured Parts](https://mlsanity.com/paper/2410.14770/) (2024)
- [Manual-PA: Learning 3D Part Assembly from Instruction Diagrams](https://mlsanity.com/paper/2411.18011/) (2024)
- [AssemblyComplete: 3D Combinatorial Construction with Deep Reinforcement Learning](https://mlsanity.com/paper/2410.15469/) (2024)
- [G-FARS: Gradient-Field-based Auto-Regressive Sampling for 3D Part Grouping](https://mlsanity.com/paper/2405.06828/) (2024)
- [PReP: Efficient context-based shape retrieval for missing parts](https://mlsanity.com/paper/2410.14245/) (2024)
- [Leveraging Pretrained Diffusion Models for Zero-Shot Part Assembly](https://mlsanity.com/paper/2505.00426/) (2025)
- [PVRA: A Pointwise Key-point Voting Framework for Robotic Assembly](https://mlsanity.com/paper/2608.19968/) (2026)
- [AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents](https://mlsanity.com/paper/2609.40353/) (2026)
- [ScaffoldM3C: A Multimodal Sequential Monte Carlo Framework for Generative Stable Construction Planning](https://mlsanity.com/paper/2610.00487/) (2026)
- [Automatic assembly modeling through attention-based assembly feature mate prediction and assembly feature pattern recognition](https://mlsanity.com/paper/2610.14633/) (2026)
- [LLM2R: From Large Language Models to 3D Reassemblers](https://mlsanity.com/paper/2610.18604/) (2026)

## Abstract (arXiv)

It is desirable to enable robots capable of automatic assembly. Structural understanding of object parts plays a crucial role in this task yet remains relatively unexplored. In this paper, we focus on the setting of furniture assembly from a complete set of part geometries, which is essentially a 6-DoF part pose estimation problem. We propose a multi-layer transformer-based framework that involves geometric and relational reasoning between parts to update the part poses iteratively. We carefully design a unique instance encoding to solve the ambiguity between geometrically-similar parts so that all parts can be distinguished. In addition to assembling from scratch, we extend our framework to a new task called in-process part assembly. Analogous to furniture maintenance, it requires robots to continue with unfinished products and assemble the remaining parts into appropriate positions. Our method achieves far more than 10% improvements over the current state-of-the-art in multiple metrics on the public PartNet dataset. Extensive experiments and quantitative comparisons demonstrate the effectiveness of the proposed framework.