# Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2110.15481 · HTML: https://mlsanity.com/paper/2110.15481/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 29 Oct 2021.

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [Junction Tree Variational Autoencoder for Molecular Graph Generation](https://mlsanity.com/paper/1802.04364/) (2018)
- [Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling](https://mlsanity.com/paper/1610.07584/) (2016)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [PointFlow: 3D Point Cloud Generation with Continuous Normalizing Flows](https://mlsanity.com/paper/1906.12320/) (2019)
- [GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models](https://mlsanity.com/paper/1802.08773/) (2018)
- [MolGAN: An implicit generative model for small molecular graphs](https://mlsanity.com/paper/1805.11973/) (2018)
- [PolyGen: An Autoregressive Generative Model of 3D Meshes](https://mlsanity.com/paper/2002.10880/) (2020)
- [Combinatorial optimization and reasoning with graph neural networks](https://mlsanity.com/paper/2102.09544/) (2021)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [Efficient Graph Generation with Graph Recurrent Attention Networks](https://mlsanity.com/paper/1910.00760/) (2019)
- [Learning Deep Generative Models of Graphs](https://mlsanity.com/paper/1803.03324/) (2018)
- [Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning](https://mlsanity.com/paper/1809.02121/) (2018)
- [A Generative Model for Molecular Distance Geometry](https://mlsanity.com/paper/1909.11459/) (2019)
- [Synthesizing Programs for Images using Reinforced Adversarial Learning](https://mlsanity.com/paper/1804.01118/) (2018)
- [Learning to Paint With Model-based Deep Reinforcement Learning](https://mlsanity.com/paper/1903.04411/) (2019)
- [NeVAE: A Deep Generative Model for Molecular Graphs](https://mlsanity.com/paper/1802.05283/) (2018)
- [Learning 3D Part Assembly from a Single Image](https://mlsanity.com/paper/2003.09754/) (2020)
- [Reinforcement Learning for Molecular Design Guided by Quantum Mechanics](https://mlsanity.com/paper/2002.07717/) (2020)
- [Structured agents for physical construction](https://mlsanity.com/paper/1904.03177/) (2019)
- [Relational inductive bias for physical construction in humans and machines](https://mlsanity.com/paper/1806.01203/) (2018)
- [Building LEGO Using Deep Generative Models of Graphs](https://mlsanity.com/paper/2012.11543/) (2020)
- [Combinatorial 3D Shape Generation via Sequential Assembly](https://mlsanity.com/paper/2004.07414/) (2020)
- [SPARE3D: A Dataset for SPAtial REasoning on Three-View Line Drawings](https://mlsanity.com/paper/2003.14034/) (2020)

## Built on by (most cited first, 19 of 19)

- [Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning](https://mlsanity.com/paper/2203.13733/) (2022)
- [IKEA-Manual: Seeing Shape Assembly Step by Step](https://mlsanity.com/paper/2302.01881/) (2023)
- [Translating a Visual LEGO Manual to a Machine-Executable Plan](https://mlsanity.com/paper/2207.12572/) (2022)
- [Break and Make: Interactive Structural Understanding Using LEGO Bricks](https://mlsanity.com/paper/2207.13738/) (2022)
- [Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Images](https://mlsanity.com/paper/2404.16423/) (2024)
- [Physics-Aware Combinatorial Assembly Sequence Planning using Data-free Action Masking](https://mlsanity.com/paper/2408.10162/) (2024)
- [BrickNet: Graph-Backed Generative Brick Assembly](https://mlsanity.com/paper/2604.22984/) (2026)
- [Budget-Aware Sequential Brick Assembly with Efficient Constraint Satisfaction](https://mlsanity.com/paper/2210.01021/) (2022)
- [A Lightweight and Transferable Design for Robust LEGO Manipulation](https://mlsanity.com/paper/2309.02354/) (2023)
- [Learning to Assemble Geometric Shapes](https://mlsanity.com/paper/2205.11809/) (2022)
- [Stable and Consistent Prediction of 3D Characteristic Orientation via Invariant Residual Learning](https://mlsanity.com/paper/2306.11406/) (2023)
- [CNC-Net: Self-Supervised Learning for CNC Machining Operations](https://mlsanity.com/paper/2312.09925/) (2023)
- [SCANet: Correcting LEGO Assembly Errors with Self-Correct Assembly Network](https://mlsanity.com/paper/2403.18195/) (2024)
- [TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assembly](https://mlsanity.com/paper/2407.15648/) (2024)
- [Learning to Build by Building Your Own Instructions](https://mlsanity.com/paper/2410.01111/) (2024)
- [Rollback-Free Stable Brick Structures Generation](https://mlsanity.com/paper/2605.06947/) (2026)
- [AssemblyComplete: 3D Combinatorial Construction with Deep Reinforcement Learning](https://mlsanity.com/paper/2410.15469/) (2024)
- [EUPHORIA: Efficient Universal Planning via Hybrid Optimization for Robust Industrial Robotic Assembly](https://mlsanity.com/paper/2605.18872/) (2026)
- [RL-Lock: Reinforcement Learning for Generating Interlocking Assemblies](https://mlsanity.com/paper/2608.01744/) (2026)

## Abstract (arXiv)

Discovering a solution in a combinatorial space is prevalent in many real-world problems but it is also challenging due to diverse complex constraints and the vast number of possible combinations. To address such a problem, we introduce a novel formulation, combinatorial construction, which requires a building agent to assemble unit primitives (i.e., LEGO bricks) sequentially -- every connection between two bricks must follow a fixed rule, while no bricks mutually overlap. To construct a target object, we provide incomplete knowledge about the desired target (i.e., 2D images) instead of exact and explicit volumetric information to the agent. This problem requires a comprehensive understanding of partial information and long-term planning to append a brick sequentially, which leads us to employ reinforcement learning. The approach has to consider a variable-sized action space where a large number of invalid actions, which would cause overlap between bricks, exist. To resolve these issues, our model, dubbed Brick-by-Brick, adopts an action validity prediction network that efficiently filters invalid actions for an actor-critic network. We demonstrate that the proposed method successfully learns to construct an unseen object conditioned on a single image or multiple views of a target object.