# IKEA-Manual: Seeing Shape Assembly Step by Step

arXiv: https://arxiv.org/abs/2302.01881 · HTML: https://mlsanity.com/paper/2302.01881/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 55 references that are in the map.
- First posted on arXiv 3 Feb 2023.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [ABC: A Big CAD Model Dataset For Geometric Deep Learning](https://mlsanity.com/paper/1812.06216/) (2018)
- [BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth](https://mlsanity.com/paper/1703.10896/) (2017)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [Thingi10K: A Dataset of 10,000 3D-Printing Models](https://mlsanity.com/paper/1605.04797/) (2016)
- [Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency](https://mlsanity.com/paper/1704.06254/) (2017)
- [JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints](https://mlsanity.com/paper/2111.12772/) (2021)
- [IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks](https://mlsanity.com/paper/1911.07246/) (2019)
- [Generative 3D Part Assembly via Dynamic Graph Learning](https://mlsanity.com/paper/2006.07793/) (2020)
- [Factory: Fast Contact for Robotic Assembly](https://mlsanity.com/paper/2205.03532/) (2022)
- [Learning 3D Part Assembly from a Single Image](https://mlsanity.com/paper/2003.09754/) (2020)
- [Structured agents for physical construction](https://mlsanity.com/paper/1904.03177/) (2019)
- [Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects](https://mlsanity.com/paper/1906.05105/) (2019)
- [RGL-NET: A Recurrent Graph Learning framework for Progressive Part Assembly](https://mlsanity.com/paper/2107.12859/) (2021)
- [Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning](https://mlsanity.com/paper/2203.13733/) (2022)
- [Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning](https://mlsanity.com/paper/2110.15481/) (2021)
- [IKEA Object State Dataset: A 6DoF object pose estimation dataset and benchmark for multi-state assembly objects](https://mlsanity.com/paper/2111.08614/) (2021)

## Built on by (most cited first, 22 of 22)

- [ZeroPS: High-quality Cross-modal Knowledge Transfer for Zero-Shot 3D Part Segmentation](https://mlsanity.com/paper/2311.14262/) (2023)
- [3D Part Segmentation via Geometric Aggregation of 2D Visual Features](https://mlsanity.com/paper/2412.04247/) (2024)
- [Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Models](https://mlsanity.com/paper/2502.10090/) (2025)
- [Rectified Point Flow: Generic Point Cloud Pose Estimation](https://mlsanity.com/paper/2506.05282/) (2025)
- [Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulation](https://mlsanity.com/paper/2504.06961/) (2025)
- [SPAFormer: Sequential 3D Part Assembly with Transformers](https://mlsanity.com/paper/2403.05874/) (2024)
- [Part2Object: Hierarchical Unsupervised 3D Instance Segmentation](https://mlsanity.com/paper/2407.10084/) (2024)
- [IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos](https://mlsanity.com/paper/2411.11409/) (2024)
- [3x2: 3D Object Part Segmentation by 2D Semantic Correspondences](https://mlsanity.com/paper/2407.09648/) (2024)
- [GBOT: Graph-Based 3D Object Tracking for Augmented Reality-Assisted Assembly Guidance](https://mlsanity.com/paper/2402.07677/) (2024)
- [Manual-PA: Learning 3D Part Assembly from Instruction Diagrams](https://mlsanity.com/paper/2411.18011/) (2024)
- [ASDF: Assembly State Detection Utilizing Late Fusion by Integrating 6D Pose Estimation](https://mlsanity.com/paper/2403.16400/) (2024)
- [BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly](https://mlsanity.com/paper/2506.06221/) (2025)
- [Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Models](https://mlsanity.com/paper/2510.16344/) (2025)
- [SCANet: Correcting LEGO Assembly Errors with Self-Correct Assembly Network](https://mlsanity.com/paper/2403.18195/) (2024)
- [Retrieval-Augmented Robots via Retrieve-Reason-Act](https://mlsanity.com/paper/2603.02688/) (2026)
- [AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects](https://mlsanity.com/paper/2605.12845/) (2026)
- [OmniCAD: A Large-Scale Benchmark for 3D Spatial Reasoning in Robotics Assemblies](https://mlsanity.com/paper/2608.22637/) (2026)
- [Query-Centric Diffusion Policy for Generalizable Robotic Assembly](https://mlsanity.com/paper/2509.18686/) (2025)
- [PinPoint3D: Fine-Grained 3D Part Segmentation from a Few Clicks](https://mlsanity.com/paper/2509.25970/) (2025)
- [Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment](https://mlsanity.com/paper/2604.00913/) (2026)
- [AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents](https://mlsanity.com/paper/2609.40353/) (2026)

## Abstract (arXiv)

Human-designed visual manuals are crucial components in shape assembly activities. They provide step-by-step guidance on how we should move and connect different parts in a convenient and physically-realizable way. While there has been an ongoing effort in building agents that perform assembly tasks, the information in human-design manuals has been largely overlooked. We identify that this is due to 1) a lack of realistic 3D assembly objects that have paired manuals and 2) the difficulty of extracting structured information from purely image-based manuals. Motivated by this observation, we present IKEA-Manual, a dataset consisting of 102 IKEA objects paired with assembly manuals. We provide fine-grained annotations on the IKEA objects and assembly manuals, including decomposed assembly parts, assembly plans, manual segmentation, and 2D-3D correspondence between 3D parts and visual manuals. We illustrate the broad application of our dataset on four tasks related to shape assembly: assembly plan generation, part segmentation, pose estimation, and 3D part assembly.