# Multi-level Reasoning for Robotic Assembly: From Sequence Inference to Contact Selection

arXiv: https://arxiv.org/abs/2312.10571 · HTML: https://mlsanity.com/paper/2312.10571/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 17 Dec 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation](https://mlsanity.com/paper/2305.12821/) (2023)
- [JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints](https://mlsanity.com/paper/2111.12772/) (2021)
- [Generative 3D Part Assembly via Dynamic Graph Learning](https://mlsanity.com/paper/2006.07793/) (2020)
- [Factory: Fast Contact for Robotic Assembly](https://mlsanity.com/paper/2205.03532/) (2022)
- [Learning 3D Part Assembly from a Single Image](https://mlsanity.com/paper/2003.09754/) (2020)
- [Tactile-RL for Insertion: Generalization to Objects of Unknown Geometry](https://mlsanity.com/paper/2104.01167/) (2021)
- [Neural Shape Mating: Self-Supervised Object Assembly with Adversarial Shape Priors](https://mlsanity.com/paper/2205.14886/) (2022)
- [Offline Meta-Reinforcement Learning for Industrial Insertion](https://mlsanity.com/paper/2110.04276/) (2021)
- [Learning Dexterous Manipulation from Exemplar Object Trajectories and Pre-Grasps](https://mlsanity.com/paper/2209.11221/) (2022)
- [Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning](https://mlsanity.com/paper/2203.13733/) (2022)
- [Learning Robotic Assembly from CAD](https://mlsanity.com/paper/1803.07635/) (2018)
- [Learning Insertion Primitives with Discrete-Continuous Hybrid Action Space for Robotic Assembly Tasks](https://mlsanity.com/paper/2110.12618/) (2021)
- [RoboAssembly: Learning Generalizable Furniture Assembly Policy in a Novel Multi-robot Contact-rich Simulation Environment](https://mlsanity.com/paper/2112.10143/) (2021)
- [6-DoF Contrastive Grasp Proposal Network](https://mlsanity.com/paper/2103.15995/) (2021)
- [Rearrangement Planning for General Part Assembly](https://mlsanity.com/paper/2307.00206/) (2023)
- [Planning Assembly Sequence with Graph Transformer](https://mlsanity.com/paper/2210.05236/) (2022)
- [Contact Pose Identification for Peg-in-Hole Assembly under Uncertainties](https://mlsanity.com/paper/2101.12467/) (2021)
- [Learn to Grasp with Less Supervision: A Data-Efficient Maximum Likelihood Grasp Sampling Loss](https://mlsanity.com/paper/2110.01379/) (2021)
- [Allowing Safe Contact in Robotic Goal-Reaching: Planning and Tracking in Operational and Null Spaces](https://mlsanity.com/paper/2211.08199/) (2022)
- [Optimization Model for Planning Precision Grasps with Multi-Fingered Hands](https://mlsanity.com/paper/1904.07332/) (2019)

## Built on by (most cited first, 6 of 6)

- [Manual-PA: Learning 3D Part Assembly from Instruction Diagrams](https://mlsanity.com/paper/2411.18011/) (2024)
- [Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Models](https://mlsanity.com/paper/2511.02162/) (2025)
- [AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects](https://mlsanity.com/paper/2605.12845/) (2026)
- [Subassembly to Full Assembly: Effective Assembly Sequence Planning through Graph-based Reinforcement Learning](https://mlsanity.com/paper/2409.13620/) (2024)
- [Learning Generalizable 3D Manipulation With 10 Demonstrations](https://mlsanity.com/paper/2411.10203/) (2024)
- [PVRA: A Pointwise Key-point Voting Framework for Robotic Assembly](https://mlsanity.com/paper/2608.19968/) (2026)

## Abstract (arXiv)

Automating the assembly of objects from their parts is a complex problem with innumerable applications in manufacturing, maintenance, and recycling. Unlike existing research, which is limited to target segmentation, pose regression, or using fixed target blueprints, our work presents a holistic multi-level framework for part assembly planning consisting of part assembly sequence inference, part motion planning, and robot contact optimization. We present the Part Assembly Sequence Transformer (PAST) -- a sequence-to-sequence neural network -- to infer assembly sequences recursively from a target blueprint. We then use a motion planner and optimization to generate part movements and contacts. To train PAST, we introduce D4PAS: a large-scale Dataset for Part Assembly Sequences (D4PAS) consisting of physically valid sequences for industrial objects. Experimental results show that our approach generalizes better than prior methods while needing significantly less computational time for inference.