mlsanityData to 7 Oct 2026

BrickNet: Graph-Backed Generative Brick Assembly

arXiv 2604.22984 ↗First posted 24 Apr 2026cs.CV cs.GRcited by 5

Not on the map. Why?

Abstract (arXiv)

We train a language model to generate LEGO-brick build sequences. While prior work has been restricted to discrete, voxel-like towers, we consider a much broader set of pieces, encompassing thousands of part types with diverse connection semantics. To enable this, we first collect a large-scale dataset of over 100,000 human-designed LDraw brick objects and scenes. The complexity of our setting makes it challenging to autoregressively assemble structures that satisfy physical constraints. When predicting block pose directly, build sequences quickly become invalid after a small number of steps. Although pieces are placed in 3D space, it is the spatial relationships of the parts which define the whole. With this in mind, we design a graph-based program representation that parametrizes structure through connectivity, improving the physical grounding of generated sequences. To enable future applications, we make our dataset and models available for research purposes. https://kulits.github.io/BrickNet

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 24 Apr 2026.

Builds on

23 of 36 references.

References · 23

  1. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  2. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  3. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  4. 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 257
  5. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  6. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  7. 2024SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Modelcited by 95
  8. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  9. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  10. 2025Generating Physically Stable and Buildable Brick Structures from Textcited by 23
  11. 2022Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learningcited by 21
  12. 2024Re-Thinking Inverse Graphics With Large Language Modelscited by 20
11 more
  1. 2021Brick-by-Brick: Combinatorial Construction with Deep Reinforcement Learningcited by 19
  2. 2022Translating a Visual LEGO Manual to a Machine-Executable Plancited by 15
  3. 2020Building LEGO Using Deep Generative Models of Graphscited by 14
  4. 2020Combinatorial 3D Shape Generation via Sequential Assemblycited by 13
  5. 2022Break and Make: Interactive Structural Understanding Using LEGO Brickscited by 8
  6. 2024Reconstructing Animals and the Wildcited by 7
  7. 2023Generating Context-Aware Natural Answers for Questions in 3D Scenescited by 5
  8. 2022Budget-Aware Sequential Brick Assembly with Efficient Constraint Satisfactioncited by 4
  9. 2024TreeSBA: Tree-Transformer for Self-Supervised Sequential Brick Assemblycited by 3
  10. 2024Learning to Build by Building Your Own Instructionscited by 3
  11. 2023A Parallel Feature-preserving Mesh Variable Offsetting Method with Dynamic Programmingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenizationcited by 2
  2. 2026Sample-Efficient Post-Training for LEGO Spatial-Physics Reasoningcited by 1
  3. 2026ResemBrick: Brick Reconstruction from Photographs with Perceptual Fidelity and Buildabilitycited by 1
  4. 2026Brick-Composer: Using MLLMs for Assembly with Diverse Brickscited by 0
  5. 2026ScaffoldM3C: A Multimodal Sequential Monte Carlo Framework for Generative Stable Construction Planningcited by 0