# Generating Physically Stable and Buildable Brick Structures from Text

arXiv: https://arxiv.org/abs/2505.05469 · HTML: https://mlsanity.com/paper/2505.05469/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 72 of its 96 references that are in the map.
- First posted on arXiv 8 May 2025.

## Builds on (most cited first, 30 of 72)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Magic3D: High-Resolution Text-to-3D Content Creation](https://mlsanity.com/paper/2211.10440/) (2022)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation](https://mlsanity.com/paper/2402.05054/) (2024)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation](https://mlsanity.com/paper/2303.13873/) (2023)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures](https://mlsanity.com/paper/2211.07600/) (2022)
- [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023)
- [CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets](https://mlsanity.com/paper/2406.13897/) (2024)
- [Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model](https://mlsanity.com/paper/2311.06214/) (2023)
- [PhysDiff: Physics-Guided Human Motion Diffusion Model](https://mlsanity.com/paper/2212.02500/) (2022)
- [Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation](https://mlsanity.com/paper/2501.12202/) (2025)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [TEXTure: Text-Guided Texturing of 3D Shapes](https://mlsanity.com/paper/2302.01721/) (2023)
- [3D Neural Field Generation using Triplane Diffusion](https://mlsanity.com/paper/2211.16677/) (2022)
- [PolyGen: An Autoregressive Generative Model of 3D Meshes](https://mlsanity.com/paper/2002.10880/) (2020)

## Built on by (most cited first, 24 of 24)

- [Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language Models](https://mlsanity.com/paper/2606.02580/) (2026)
- [PhysInOne: Visual Physics Learning and Reasoning in One Suite](https://mlsanity.com/paper/2604.09415/) (2026)
- [Engine-Native Editable 3D World Reconstruction with Objects and Lighting](https://mlsanity.com/paper/2607.20889/) (2026)
- [Learn2Fold: Structured Origami Generation with World Model Planning](https://mlsanity.com/paper/2603.29585/) (2026)
- [LottieGPT: Tokenizing Vector Animation for Autoregressive Generation](https://mlsanity.com/paper/2604.11792/) (2026)
- [BrickNet: Graph-Backed Generative Brick Assembly](https://mlsanity.com/paper/2604.22984/) (2026)
- [MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMs](https://mlsanity.com/paper/2607.08970/) (2026)
- [CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models](https://mlsanity.com/paper/2601.21798/) (2026)
- [MonoPhysics: Estimating Geometry, Appearance, and Physical Parameters from Monocular Videos](https://mlsanity.com/paper/2605.30320/) (2026)
- [HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactions](https://mlsanity.com/paper/2603.15612/) (2026)
- [Know3D: Prompting 3D Generation with Knowledge from Vision-Language Models](https://mlsanity.com/paper/2603.22782/) (2026)
- [Rollback-Free Stable Brick Structures Generation](https://mlsanity.com/paper/2605.06947/) (2026)
- [AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects](https://mlsanity.com/paper/2605.12845/) (2026)
- [BrickAnything: Geometry-Conditioned Buildable Brick Generation with Structure-Aware Tokenization](https://mlsanity.com/paper/2605.26182/) (2026)
- [Prompt-to-Product: Generative Assembly via Bimanual Manipulation](https://mlsanity.com/paper/2508.21063/) (2025)
- [Designing Tools with Control Confidence](https://mlsanity.com/paper/2510.12630/) (2025)
- [SuperVoxelGPT: Adaptive and Ordered 3D Tokenization for Autoregressive Shape Generation](https://mlsanity.com/paper/2605.29655/) (2026)
- [ResemBrick: Brick Reconstruction from Photographs with Perceptual Fidelity and Buildability](https://mlsanity.com/paper/2608.09597/) (2026)
- [CompoSE: Compositional Synthesis and Editing of 3D Shapes via Part-Aware Control](https://mlsanity.com/paper/2605.19350/) (2026)
- [Brick-Composer: Using MLLMs for Assembly with Diverse Bricks](https://mlsanity.com/paper/2606.05445/) (2026)
- [COrigami: An AI Pipeline for Co-Designing Flat-Foldable Visually Recognisable Origami](https://mlsanity.com/paper/2606.26299/) (2026)
- [PART: Learning 3D Part Assembly and Retrieval with Transformers](https://mlsanity.com/paper/2609.19872/) (2026)
- [Text2Sim: Agentic Physics-Based Simulation Generation with Distilled Expertise](https://mlsanity.com/paper/2609.36593/) (2026)
- [LLM2R: From Large Language Models to 3D Reassemblers](https://mlsanity.com/paper/2610.18604/) (2026)

## Abstract (arXiv)

We introduce BrickGPT, the first approach for generating physically stable interconnecting brick assembly models from text prompts. To achieve this, we construct a large-scale, physically stable dataset of brick structures, along with their associated captions, and train an autoregressive large language model to predict the next brick to add via next-token prediction. To improve the stability of the resulting designs, we employ an efficient validity check and physics-aware rollback during autoregressive inference, which prunes infeasible token predictions using physics laws and assembly constraints. Our experiments show that BrickGPT produces stable, diverse, and aesthetically pleasing brick structures that align closely with the input text prompts. We also develop a text-based brick texturing method to generate colored and textured designs. We show that our designs can be assembled manually by humans and automatically by robotic arms. We release our new dataset, StableText2Brick, containing over 47,000 brick structures of over 28,000 unique 3D objects accompanied by detailed captions, along with our code and models at the project website: https://avalovelace1.github.io/BrickGPT/.