# Simulation-aided Learning from Demonstration for Robotic LEGO Construction

arXiv: https://arxiv.org/abs/2309.11010 · HTML: https://mlsanity.com/paper/2309.11010/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2023.

## Built on by (most cited first, 4 of 4)

- [Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts](https://mlsanity.com/paper/2409.03966/) (2024)
- [Physics-Aware Combinatorial Assembly Sequence Planning using Data-free Action Masking](https://mlsanity.com/paper/2408.10162/) (2024)
- [AssemblyComplete: 3D Combinatorial Construction with Deep Reinforcement Learning](https://mlsanity.com/paper/2410.15469/) (2024)
- [ScaffoldM3C: A Multimodal Sequential Monte Carlo Framework for Generative Stable Construction Planning](https://mlsanity.com/paper/2610.00487/) (2026)

## Abstract (arXiv)

Recent advancements in manufacturing have a growing demand for fast, automatic prototyping (i.e. assembly and disassembly) capabilities to meet users' needs. This paper studies automatic rapid LEGO prototyping, which is devoted to constructing target LEGO objects that satisfy individual customization needs and allow users to freely construct their novel designs. A construction plan is needed in order to automatically construct the user-specified LEGO design. However, a freely designed LEGO object might not have an existing construction plan, and generating such a LEGO construction plan requires a non-trivial effort since it requires accounting for numerous constraints (e.g. object shape, colors, stability, etc.). In addition, programming the prototyping skill for the robot requires the users to have expert programming skills, which makes the task beyond the reach of the general public. To address the challenges, this paper presents a simulation-aided learning from demonstration (SaLfD) framework for easily deploying LEGO prototyping capability to robots. In particular, the user demonstrates constructing the customized novel LEGO object. The robot extracts the task information by observing the human operation and generates the construction plan. A simulation is developed to verify the correctness of the learned construction plan and the resulting LEGO prototype. The proposed system is deployed to a FANUC LR-mate 200id/7L robot. Experiments demonstrate that the proposed SaLfD framework can effectively correct and learn the prototyping (i.e. assembly and disassembly) tasks from human demonstrations. And the learned prototyping tasks are realized by the FANUC robot.