# Prompt-to-Parts: Generative AI for Physical Assembly and Scalable Instructions

arXiv: https://arxiv.org/abs/2512.15743 · HTML: https://mlsanity.com/paper/2512.15743/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 10 Dec 2025.

## Builds on (most cited first, 26 of 26)

- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL](https://mlsanity.com/paper/2406.08426/) (2024)
- [Physics-Informed Diffusion Models](https://mlsanity.com/paper/2403.14404/) (2024)
- [CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM](https://mlsanity.com/paper/2411.04954/) (2024)
- [Physics-Constrained Flow Matching: Sampling Generative Models with Hard Constraints](https://mlsanity.com/paper/2506.04171/) (2025)
- [LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?](https://mlsanity.com/paper/2503.19990/) (2025)
- [Text-to-CadQuery: A New Paradigm for CAD Generation with Scalable Large Model Capabilities](https://mlsanity.com/paper/2505.06507/) (2025)
- [MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks](https://mlsanity.com/paper/2410.10563/) (2024)
- [Physically Compatible 3D Object Modeling from a Single Image](https://mlsanity.com/paper/2405.20510/) (2024)
- [WorldScribe: Towards Context-Aware Live Visual Descriptions](https://mlsanity.com/paper/2408.06627/) (2024)
- [DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundness](https://mlsanity.com/paper/2503.22677/) (2025)
- [CADmium: Fine-Tuning Code Language Models for Text-Driven Sequential CAD Design](https://mlsanity.com/paper/2507.09792/) (2025)
- [Image2Lego: Customized LEGO Set Generation from Images](https://mlsanity.com/paper/2108.08477/) (2021)
- [From REST to MCP: An Empirical Study of API Wrapping and Automated Server Generation for LLM Agents](https://mlsanity.com/paper/2507.16044/) (2025)
- [ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulation](https://mlsanity.com/paper/2512.02013/) (2025)
- [SPAFormer: Sequential 3D Part Assembly with Transformers](https://mlsanity.com/paper/2403.05874/) (2024)
- [Break and Make: Interactive Structural Understanding Using LEGO Bricks](https://mlsanity.com/paper/2207.13738/) (2022)
- [Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality](https://mlsanity.com/paper/2405.13034/) (2024)
- [Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Images](https://mlsanity.com/paper/2404.16423/) (2024)
- [LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants](https://mlsanity.com/paper/2507.05515/) (2025)
- [Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Models](https://mlsanity.com/paper/2510.16344/) (2025)
- [Learning to Build by Building Your Own Instructions](https://mlsanity.com/paper/2410.01111/) (2024)
- [GLLM: Self-Corrective G-Code Generation using Large Language Models with User Feedback](https://mlsanity.com/paper/2501.17584/) (2025)
- [NarraGuide: an LLM-based Narrative Mobile Robot for Remote Place Exploration](https://mlsanity.com/paper/2508.01235/) (2025)
- [PLUGH: A Benchmark for Spatial Understanding and Reasoning in Large Language Models](https://mlsanity.com/paper/2408.04648/) (2024)
- [BrickSmart: Leveraging Generative AI to Support Children's Spatial Language Learning in Family Block Play](https://mlsanity.com/paper/2504.11138/) (2025)

## Abstract (arXiv)

We present a framework for generating physically realizable assembly instructions from natural language descriptions. Unlike unconstrained text-to-3D approaches, our method operates within a discrete parts vocabulary, enforcing geometric validity, connection constraints, and buildability ordering. Using LDraw as a text-rich intermediate representation, we demonstrate that large language models can be guided with tools to produce valid step-by-step construction sequences and assembly instructions for brick-based prototypes of more than 3000 assembly parts. We introduce a Python library for programmatic model generation and evaluate buildable outputs on complex satellites, aircraft, and architectural domains. The approach aims for demonstrable scalability, modularity, and fidelity that bridges the gap between semantic design intent and manufacturable output. Physical prototyping follows from natural language specifications. The work proposes a novel elemental lingua franca as a key missing piece from the previous pixel-based diffusion methods or computer-aided design (CAD) models that fail to support complex assembly instructions or component exchange. Across four original designs, this novel "bag of bricks" method thus functions as a physical API: a constrained vocabulary connecting precisely oriented brick locations to a "bag of words" through which arbitrary functional requirements compile into material reality. Given such a consistent and repeatable AI representation opens new design options while guiding natural language implementations in manufacturing and engineering prototyping.