# A Lightweight and Transferable Design for Robust LEGO Manipulation

arXiv: https://arxiv.org/abs/2309.02354 · HTML: https://mlsanity.com/paper/2309.02354/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 52 references that are in the map.
- First posted on arXiv 5 Sep 2023.

## Builds on (most cited first, 19 of 19)

- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Data-efficient Deep Reinforcement Learning for Dexterous Manipulation](https://mlsanity.com/paper/1704.03073/) (2017)
- [Warm Starting CMA-ES for Hyperparameter Optimization](https://mlsanity.com/paper/2012.06932/) (2020)
- [Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning](https://mlsanity.com/paper/2110.15481/) (2021)
- [Safe Control Algorithms Using Energy Functions: A Unified Framework, Benchmark, and New Directions](https://mlsanity.com/paper/1908.01883/) (2019)
- [Translating a Visual LEGO Manual to a Machine-Executable Plan](https://mlsanity.com/paper/2207.12572/) (2022)
- [Building LEGO Using Deep Generative Models of Graphs](https://mlsanity.com/paper/2012.11543/) (2020)
- [Image2Lego: Customized LEGO Set Generation from Images](https://mlsanity.com/paper/2108.08477/) (2021)
- [Combinatorial 3D Shape Generation via Sequential Assembly](https://mlsanity.com/paper/2004.07414/) (2020)
- [GUARD: A Safe Reinforcement Learning Benchmark](https://mlsanity.com/paper/2305.13681/) (2023)
- [Break and Make: Interactive Structural Understanding Using LEGO Bricks](https://mlsanity.com/paper/2207.13738/) (2022)
- [A Learning Framework for High Precision Industrial Assembly](https://mlsanity.com/paper/1809.08548/) (2018)
- [Budget-Aware Sequential Brick Assembly with Efficient Constraint Satisfaction](https://mlsanity.com/paper/2210.01021/) (2022)
- [Proactive Human-Robot Co-Assembly: Leveraging Human Intention Prediction and Robust Safe Control](https://mlsanity.com/paper/2306.11862/) (2023)
- [Safe Interactive Industrial Robots using Jerk-based Safe Set Algorithm](https://mlsanity.com/paper/2204.03038/) (2022)
- [Task-Agnostic Adaptation for Safe Human-Robot Handover](https://mlsanity.com/paper/2209.09418/) (2022)
- [Zero-shot Transferable and Persistently Feasible Safe Control for High Dimensional Systems by Consistent Abstraction](https://mlsanity.com/paper/2303.10277/) (2023)
- [Robotic LEGO Assembly and Disassembly from Human Demonstration](https://mlsanity.com/paper/2305.15667/) (2023)
- [A Composable Framework for Policy Design, Learning, and Transfer Toward Safe and Efficient Industrial Insertion](https://mlsanity.com/paper/2203.03017/) (2022)

## Built on by (most cited first, 4 of 4)

- [Generating Physically Stable and Buildable Brick Structures from Text](https://mlsanity.com/paper/2505.05469/) (2025)
- [Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts](https://mlsanity.com/paper/2409.03966/) (2024)
- [Physics-Aware Combinatorial Assembly Sequence Planning using Data-free Action Masking](https://mlsanity.com/paper/2408.10162/) (2024)
- [Prompt-to-Product: Generative Assembly via Bimanual Manipulation](https://mlsanity.com/paper/2508.21063/) (2025)

## Abstract (arXiv)

Lego is a well-known platform for prototyping pixelized objects. However, robotic Lego prototyping (i.e., manipulating Lego bricks) is challenging due to the tight connections and accuracy requirements. This paper investigates safe and efficient robotic Lego manipulation. In particular, this paper reduces the complexity of the manipulation by hardware-software co-design. An end-of-arm tool (EOAT) is designed, which reduces the problem dimension and allows large industrial robots to manipulate small Lego bricks. In addition, this paper uses evolution strategy to optimize the robot motion for Lego manipulation. Experiments demonstrate that the EOAT can reliably manipulate Lego bricks and the learning framework can effectively and safely improve the manipulation performance to a 100% success rate. The co-design is deployed to multiple robots (i.e., FANUC LR-mate 200id/7L and Yaskawa GP4) to demonstrate its generalizability and transferability. In the end, we show that the proposed solution enables sustainable robotic Lego prototyping, in which the robot can repeatedly assemble and disassemble different prototypes.