# Speech to Reality: On-Demand Production using Natural Language, 3D Generative AI, and Discrete Robotic Assembly

arXiv: https://arxiv.org/abs/2409.18390 · HTML: https://mlsanity.com/paper/2409.18390/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 44 references that are in the map.
- First posted on arXiv 27 Sep 2024.

## Builds on (most cited first, 16 of 16)

- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Neuralangelo: High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2306.03092/) (2023)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [Shap-E: Generating Conditional 3D Implicit Functions](https://mlsanity.com/paper/2305.02463/) (2023)
- [Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials](https://mlsanity.com/paper/2407.02445/) (2024)
- [Generative AI meets 3D: A Survey on Text-to-3D in AIGC Era](https://mlsanity.com/paper/2305.06131/) (2023)
- [How Can Large Language Models Help Humans in Design and Manufacturing?](https://mlsanity.com/paper/2307.14377/) (2023)
- [LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis](https://mlsanity.com/paper/2403.15385/) (2024)
- [An Automated Survey of Generative Artificial Intelligence: Large Language Models, Architectures, Protocols, and Applications](https://mlsanity.com/paper/2306.02781/) (2023)
- [ASAP: Automated Sequence Planning for Complex Robotic Assembly with Physical Feasibility](https://mlsanity.com/paper/2309.16909/) (2023)
- [Sketch2Prototype: Rapid Conceptual Design Exploration and Prototyping with Generative AI](https://mlsanity.com/paper/2405.12985/) (2024)
- [Style2Fab: Functionality-Aware Segmentation for Fabricating Personalized 3D Models with Generative AI](https://mlsanity.com/paper/2309.06379/) (2023)
- [Shaping Realities: Enhancing 3D Generative AI with Fabrication Constraints](https://mlsanity.com/paper/2404.10142/) (2024)
- [Toward Automated Programming for Robotic Assembly Using ChatGPT](https://mlsanity.com/paper/2405.08216/) (2024)
- [CADgpt: Harnessing Natural Language Processing for 3D Modelling to Enhance Computer-Aided Design Workflows](https://mlsanity.com/paper/2401.05476/) (2024)
- [AI in Manufacturing: Market Analysis and Opportunities](https://mlsanity.com/paper/2407.05426/) (2024)

## Built on by (most cited first, 4 of 4)

- [Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Models](https://mlsanity.com/paper/2511.02162/) (2025)
- [Prompt-to-Product: Generative Assembly via Bimanual Manipulation](https://mlsanity.com/paper/2508.21063/) (2025)
- [AI Assisted AR Assembly: Object Recognition and Computer Vision for Augmented Reality Assisted Assembly](https://mlsanity.com/paper/2511.05394/) (2025)
- [Augmented Assembly: Object Recognition and Hand Tracking for Adaptive Assembly Instructions in Augmented Reality](https://mlsanity.com/paper/2601.11535/) (2025)

## Abstract (arXiv)

We present a system that transforms speech into physical objects using 3D generative AI and discrete robotic assembly. By leveraging natural language, the system makes design and manufacturing more accessible to people without expertise in 3D modeling or robotic programming. While generative AI models can produce a wide range of 3D meshes, AI-generated meshes are not directly suitable for robotic assembly or account for fabrication constraints. To address this, we contribute a workflow that integrates natural language, 3D generative AI, geometric processing, and discrete robotic assembly. The system discretizes the AI-generated geometry and modifies it to meet fabrication constraints such as component count, overhangs, and connectivity to ensure feasible physical assembly. The results are demonstrated through the assembly of various objects, ranging from chairs to shelves, which are prompted via speech and realized within 5 minutes using a robotic arm.