mlsanityData to 14 Oct 2026

Translating a Visual LEGO Manual to a Machine-Executable Plan

arXiv 2207.12572 ↗First posted 25 Jul 2022cs.CV cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

We study the problem of translating an image-based, step-by-step assembly manual created by human designers into machine-interpretable instructions. We formulate this problem as a sequential prediction task: at each step, our model reads the manual, locates the components to be added to the current shape, and infers their 3D poses. This task poses the challenge of establishing a 2D-3D correspondence between the manual image and the real 3D object, and 3D pose estimation for unseen 3D objects, since a new component to be added in a step can be an object built from previous steps. To address these two challenges, we present a novel learning-based framework, the Manual-to-Executable-Plan Network (MEPNet), which reconstructs the assembly steps from a sequence of manual images. The key idea is to integrate neural 2D keypoint detection modules and 2D-3D projection algorithms for high-precision prediction and strong generalization to unseen components. The MEPNet outperforms existing methods on three newly collected LEGO manual datasets and a Minecraft house dataset.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 28 of its 52 references that are in the map. First posted on arXiv 25 Jul 2022.

Builds on

28 of 52 references.

References · 28

  1. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  2. 2015Spatial Transformer Networkscited by 1,429
  3. 2019Objects as Pointscited by 1,175
  4. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 979
  5. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 915
  6. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 775
  7. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 651
  8. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  9. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  10. 2018PVNet: Pixel-wise Voting Network for 6DoF Pose Estimationcited by 385
  11. 2019DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusioncited by 355
  12. 2018An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solutioncited by 307
16 more
  1. 2018DeepIM: Deep Iterative Matching for 6D Pose Estimationcited by 283
  2. 2017BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depthcited by 225
  3. 2016Associative Embedding: End-to-End Learning for Joint Detection and Groupingcited by 212
  4. 2016Learning Shape Abstractions by Assembling Volumetric Primitivescited by 181
  5. 2020Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wildcited by 142
  6. 2021JoinABLe: Learning Bottom-up Assembly of Parametric CAD Jointscited by 78
  7. 2018Making Deep Heatmaps Robust to Partial Occlusions for 3D Object Pose Estimationcited by 74
  8. 2019IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Taskscited by 69
  9. 2019Learning to Infer and Execute 3D Shape Programscited by 68
  10. 2020Generative 3D Part Assembly via Dynamic Graph Learningcited by 67
  11. 2018Im2Struct: Recovering 3D Shape Structure from a Single RGB Imagecited by 55
  12. 2020Learning 3D Part Assembly from a Single Imagecited by 52
  13. 2021Inferring CAD Modeling Sequences Using Zone Graphscited by 50
  14. 2019Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objectscited by 39
  15. 2020Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequencescited by 36
  16. 2021Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learningcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 2; 2026: 3.

15 papers.

Sort:

  1. 2025Generating Physically Stable and Buildable Brick Structures from Textcited by 24
  2. 2023MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devicescited by 15
  3. 2025Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Modelscited by 14
  4. 2024SPAFormer: Sequential 3D Part Assembly with Transformerscited by 12
  5. 2023Aligning Step-by-Step Instructional Diagrams to Video Demonstrationscited by 10
  6. 2024A Survey on Computational Solutions for Reconstructing Complete Objects by Reassembling Their Fractured Partscited by 9
  7. 2024Manual-PA: Learning 3D Part Assembly from Instruction Diagramscited by 7
  8. 2023DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physicscited by 6
  9. 2024Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Imagescited by 5
  10. 2026BrickNet: Graph-Backed Generative Brick Assemblycited by 5
  11. 2023A Lightweight and Transferable Design for Robust LEGO Manipulationcited by 4
  12. 2024TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assemblycited by 3
3 more
  1. 2024Learning to Build by Building Your Own Instructionscited by 3
  2. 2026AssemblyBench: Physics-Aware Assembly of Complex Industrial Objectscited by 2
  3. 2026Brick-Composer: Using MLLMs for Assembly with Diverse Brickscited by 0