# Budget-Aware Sequential Brick Assembly with Efficient Constraint Satisfaction

arXiv: https://arxiv.org/abs/2210.01021 · HTML: https://mlsanity.com/paper/2210.01021/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 18 of 18)

- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation](https://mlsanity.com/paper/1806.02473/) (2018)
- [High-Fidelity Pluralistic Image Completion with Transformers](https://mlsanity.com/paper/2103.14031/) (2021)
- [Multiresolution Tree Networks for 3D Point Cloud Processing](https://mlsanity.com/paper/1807.03520/) (2018)
- [JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints](https://mlsanity.com/paper/2111.12772/) (2021)
- [IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks](https://mlsanity.com/paper/1911.07246/) (2019)
- [Neural Shape Mating: Self-Supervised Object Assembly with Adversarial Shape Priors](https://mlsanity.com/paper/2205.14886/) (2022)
- [Learning to Generate 3D Shapes with Generative Cellular Automata](https://mlsanity.com/paper/2103.04130/) (2021)
- [Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning](https://mlsanity.com/paper/2203.13733/) (2022)
- [Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning](https://mlsanity.com/paper/2110.15481/) (2021)
- [Building LEGO Using Deep Generative Models of Graphs](https://mlsanity.com/paper/2012.11543/) (2020)
- [Combinatorial 3D Shape Generation via Sequential Assembly](https://mlsanity.com/paper/2004.07414/) (2020)
- [In silico generation of novel, drug-like chemical matter using the LSTM neural network](https://mlsanity.com/paper/1712.07449/) (2017)
- [Learning to Design and Construct Bridge without Blueprint](https://mlsanity.com/paper/2108.02439/) (2021)
- [Learning to Assemble Geometric Shapes](https://mlsanity.com/paper/2205.11809/) (2022)

## Built on by (most cited first, 4 of 4)

- [BrickNet: Graph-Backed Generative Brick Assembly](https://mlsanity.com/paper/2604.22984/) (2026)
- [A Lightweight and Transferable Design for Robust LEGO Manipulation](https://mlsanity.com/paper/2309.02354/) (2023)
- [Stable and Consistent Prediction of 3D Characteristic Orientation via Invariant Residual Learning](https://mlsanity.com/paper/2306.11406/) (2023)
- [Sample-Efficient Post-Training for LEGO Spatial-Physics Reasoning](https://mlsanity.com/paper/2606.07602/) (2026)

## Abstract (arXiv)

We tackle the problem of sequential brick assembly with LEGO bricks to create combinatorial 3D structures. This problem is challenging since this brick assembly task encompasses the characteristics of combinatorial optimization problems. In particular, the number of assemblable structures increases exponentially as the number of bricks used increases. To solve this problem, we propose a new method to predict the scores of the next brick position by employing a U-shaped sparse 3D convolutional neural network. Along with the 3D convolutional network, a one-initialized brick-sized convolution filter is used to efficiently validate assembly constraints between bricks without training itself. By the nature of this one-initialized convolution filter, we can readily consider several different brick types by benefiting from modern implementation of convolution operations. To generate a novel structure, we devise a sampling strategy to determine the next brick position considering the satisfaction of assembly constraints. Moreover, our method is designed for either budget-free or budget-aware scenario where a budget may confine the number of bricks and their types. We demonstrate that our method successfully generates a variety of brick structures and outperforms existing methods with Bayesian optimization, deep graph generative model, and reinforcement learning.