# Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Models

arXiv: https://arxiv.org/abs/2511.02162 · HTML: https://mlsanity.com/paper/2511.02162/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 4 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 23 of 23)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models](https://mlsanity.com/paper/2404.07191/) (2024)
- [Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks](https://mlsanity.com/paper/2103.10429/) (2021)
- [SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation](https://mlsanity.com/paper/1612.00606/) (2016)
- [PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Models](https://mlsanity.com/paper/2412.18608/) (2024)
- [Generative AI meets 3D: A Survey on Text-to-3D in AIGC Era](https://mlsanity.com/paper/2305.06131/) (2023)
- [StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects](https://mlsanity.com/paper/2211.04604/) (2022)
- [AutoDecoding Latent 3D Diffusion Models](https://mlsanity.com/paper/2307.05445/) (2023)
- [PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentation](https://mlsanity.com/paper/1903.00709/) (2019)
- [LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis](https://mlsanity.com/paper/2403.15385/) (2024)
- [Category-Level Multi-Part Multi-Joint 3D Shape Assembly](https://mlsanity.com/paper/2303.06163/) (2023)
- [Sketch2Prototype: Rapid Conceptual Design Exploration and Prototyping with Generative AI](https://mlsanity.com/paper/2405.12985/) (2024)
- [Style2Fab: Functionality-Aware Segmentation for Fabricating Personalized 3D Models with Generative AI](https://mlsanity.com/paper/2309.06379/) (2023)
- [Multi-level Reasoning for Robotic Assembly: From Sequence Inference to Contact Selection](https://mlsanity.com/paper/2312.10571/) (2023)
- [Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Images](https://mlsanity.com/paper/2404.16423/) (2024)
- [Speech to Reality: On-Demand Production using Natural Language, 3D Generative AI, and Discrete Robotic Assembly](https://mlsanity.com/paper/2409.18390/) (2024)
- [Making Physical Objects with Generative AI and Robotic Assembly: Considering Fabrication Constraints, Sustainability, Time, Functionality, and Accessibility](https://mlsanity.com/paper/2504.19131/) (2025)
- [Gesture Recognition for Feedback Based Mixed Reality and Robotic Fabrication: A Case Study of the UnLog Tower](https://mlsanity.com/paper/2409.19281/) (2024)
- [Hierarchical Discrete Lattice Assembly: An Approach for the Digital Fabrication of Scalable Macroscale Structures](https://mlsanity.com/paper/2510.13686/) (2025)

## Built on by (most cited first, 2 of 2)

- [Context Aware AI Assistant and AR Interface for Lunar Extravehicular Activity (EVA) Procedural Guidance](https://mlsanity.com/paper/2608.13589/) (2026)
- [Training Generalist Navigation Policies via Action Self-Distillation](https://mlsanity.com/paper/2610.17593/) (2026)

## Abstract (arXiv)

Advances in 3D generative AI have enabled the creation of physical objects from text prompts, but challenges remain in creating objects involving multiple component types. We present a pipeline that integrates 3D generative AI with vision-language models (VLMs) to enable the robotic assembly of multi-component objects from natural language. Our method leverages VLMs for zero-shot, multi-modal reasoning about geometry and functionality to decompose AI-generated meshes into multi-component 3D models using predefined structural and panel components. We demonstrate that a VLM is capable of determining which mesh regions need panel components in addition to structural components, based on the object's geometry and functionality. Evaluation across test objects shows that users preferred the VLM-generated assignments 90.6% of the time, compared to 59.4% for rule-based and 2.5% for random assignment. Lastly, the system allows users to refine component assignments through conversational feedback, enabling greater human control and agency in making physical objects with generative AI and robotics.