mlsanityData to 7 Oct 2026

Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Models

arXiv 2511.02162 ↗First posted 4 Nov 2025cs.RO cs.AI cs.HCcited by 1

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Advances in 3D generative AI have enabled the creation of physical objects from text prompts, but challenges remain in creating objects involving multiple component types. We present a pipeline that integrates 3D generative AI with vision-language models (VLMs) to enable the robotic assembly of multi-component objects from natural language. Our method leverages VLMs for zero-shot, multi-modal reasoning about geometry and functionality to decompose AI-generated meshes into multi-component 3D models using predefined structural and panel components. We demonstrate that a VLM is capable of determining which mesh regions need panel components in addition to structural components, based on the object's geometry and functionality. Evaluation across test objects shows that users preferred the VLM-generated assignments 90.6% of the time, compared to 59.4% for rule-based and 2.5% for random assignment. Lastly, the system allows users to refine component assignments through conversational feedback, enabling greater human control and agency in making physical objects with generative AI and robotics.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 30 references that are in the map. First posted on arXiv 4 Nov 2025.

Builds on

23 of 30 references.

References · 23

  1. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  2. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  5. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  6. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
  7. 2021Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networkscited by 101
  8. 2016SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentationcited by 70
  9. 2023Generative AI meets 3D: A Survey on Text-to-3D in AIGC Eracited by 61
  10. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Modelscited by 60
  11. 2023AutoDecoding Latent 3D Diffusion Modelscited by 50
  12. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 49
11 more
  1. 2019PartNet: A Recursive Part Decomposition Network for Fine-grained and Hierarchical Shape Segmentationcited by 44
  2. 2024LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesiscited by 39
  3. 2023Category-Level Multi-Part Multi-Joint 3D Shape Assemblycited by 22
  4. 2024Sketch2Prototype: Rapid Conceptual Design Exploration and Prototyping with Generative AIcited by 13
  5. 2023Style2Fab: Functionality-Aware Segmentation for Fabricating Personalized 3D Models with Generative AIcited by 11
  6. 2023Multi-level Reasoning for Robotic Assembly: From Sequence Inference to Contact Selectioncited by 5
  7. 2024Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Imagescited by 5
  8. 2024Speech to Reality: On-Demand Production using Natural Language, 3D Generative AI, and Discrete Robotic Assemblycited by 4
  9. 2025Making Physical Objects with Generative AI and Robotic Assembly: Considering Fabrication Constraints, Sustainability, Time, Functionality, and Accessibilitycited by 4
  10. 2024Gesture Recognition for Feedback Based Mixed Reality and Robotic Fabrication: A Case Study of the UnLog Towercited by 2
  11. 2025Hierarchical Discrete Lattice Assembly: An Approach for the Digital Fabrication of Scalable Macroscale Structurescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Context Aware AI Assistant and AR Interface for Lunar Extravehicular Activity (EVA) Procedural Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset
  2. 20253DFroMLLM: 3D Prototype Generation only from Pretrained Multimodal LLMs
  3. 2024Task-oriented Robotic Manipulation with Vision Language Models
  4. 2025VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangement
  5. 2025Let Language Constrain Geometry: Vision-Language Models as Semantic and Spatial Critics for 3D Generation
  6. 2025Agentic 3D Scene Generation with Spatially Contextualized VLMs
  7. 20263D Space as a Scratchpad for Editable Text-to-Image Generation
  8. 20243D-VLA: A 3D Vision-Language-Action Generative World Model