mlsanityData to 14 Oct 2026

Break and Make: Interactive Structural Understanding Using LEGO Bricks

arXiv 2207.13738 ↗First posted 27 Jul 2022cs.CV cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Visual understanding of geometric structures with complex spatial relationships is a fundamental component of human intelligence. As children, we learn how to reason about structure not only from observation, but also by interacting with the world around us -- by taking things apart and putting them back together again. The ability to reason about structure and compositionality allows us to not only build things, but also understand and reverse-engineer complex systems. In order to advance research in interactive reasoning for part-based geometric understanding, we propose a challenging new assembly problem using LEGO bricks that we call Break and Make. In this problem an agent is given a LEGO model and attempts to understand its structure by interactively inspecting and disassembling it. After this inspection period, the agent must then prove its understanding by rebuilding the model from scratch using low-level action primitives. In order to facilitate research on this problem we have built LTRON, a fully interactive 3D simulator that allows learning agents to assemble, disassemble and manipulate LEGO models. We pair this simulator with a new dataset of fan-made LEGO creations that have been uploaded to the internet in order to provide complex scenes containing over a thousand unique brick shapes. We take a first step towards solving this problem using sequence-to-sequence models that provide guidance for how to make progress on this challenging problem. Our simulator and data are available at github.com/aaronwalsman/ltron. Additional training code and PyTorch examples are available at github.com/aaronwalsman/ltron-torch-eccv22.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 34 of its 62 references that are in the map. First posted on arXiv 27 Jul 2022.

Builds on

34 of 62 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  6. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2019FCOS: Fully Convolutional One-Stage Object Detectioncited by 1,577
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  10. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  11. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  12. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
22 more
  1. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 526
  2. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  3. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  4. 2017Embodied Question Answeringcited by 336
  5. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  6. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 193
  7. 2017IQA: Visual Question Answering in Interactive Environmentscited by 155
  8. 20203D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humanscited by 144
  9. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  10. 2018Fusion++: Volumetric Object-Level SLAMcited by 98
  11. 2020MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigationcited by 95
  12. 2019IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Taskscited by 69
  13. 2020NodeSLAM: Neural Object Descriptors for Multi-View Shape Reconstructioncited by 61
  14. 2020Learning 3D Part Assembly from a Single Imagecited by 52
  15. 2019Structured agents for physical constructioncited by 50
  16. 2017Deep Reinforcement Learning for High Precision Assembly Taskscited by 39
  17. 2021The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigationcited by 27
  18. 2022Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learningcited by 22
  19. 2021Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learningcited by 19
  20. 2020Building LEGO Using Deep Generative Models of Graphscited by 14
  21. 2021Image2Lego: Customized LEGO Set Generation from Imagescited by 14
  22. 2020GeoFusion: Geometric Consistency informed Scene Estimation in Dense Cluttercited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 3; 2025: 1; 2026: 2.

8 papers.

  1. 2022Geoclidean: Few-Shot Generalization in Euclidean Geometrycited by 6
  2. 2024Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Imagescited by 5
  3. 2026BrickNet: Graph-Backed Generative Brick Assemblycited by 5
  4. 2023A Lightweight and Transferable Design for Robust LEGO Manipulationcited by 4
  5. 2026Gesturing Toward Abstraction: Multimodal Convention Formation in Collaborative Physical Taskscited by 4
  6. 2024TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assemblycited by 3
  7. 2024Learning to Build by Building Your Own Instructionscited by 3
  8. 2025Prompt-to-Parts: Generative AI for Physical Assembly and Scalable Instructionscited by 0