# Prompt2Craft: Generating Functional Craft Assemblies with LLMs

arXiv: https://arxiv.org/abs/2512.04568 · HTML: https://mlsanity.com/paper/2512.04568/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 4 Dec 2025.

## Builds on (most cited first, 25 of 25)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024)
- [TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models](https://mlsanity.com/paper/2502.06608/) (2025)
- [Physically Grounded Vision-Language Models for Robotic Manipulation](https://mlsanity.com/paper/2309.02561/) (2023)
- [GenSim: Generating Robotic Simulation Tasks via Large Language Models](https://mlsanity.com/paper/2310.01361/) (2023)
- [PartCrafter: Structured 3D Mesh Generation via Compositional Latent Diffusion Transformers](https://mlsanity.com/paper/2506.05573/) (2025)
- [JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints](https://mlsanity.com/paper/2111.12772/) (2021)
- [LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoning](https://mlsanity.com/paper/2403.11552/) (2024)
- [Learning 3D Part Assembly from a Single Image](https://mlsanity.com/paper/2003.09754/) (2020)
- [RoboGround: Robotic Manipulation with Grounded Vision-Language Priors](https://mlsanity.com/paper/2504.21530/) (2025)
- [R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation](https://mlsanity.com/paper/2505.02018/) (2025)
- [ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning](https://mlsanity.com/paper/2406.19741/) (2024)
- [Category-Level Multi-Part Multi-Joint 3D Shape Assembly](https://mlsanity.com/paper/2303.06163/) (2023)
- [Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset](https://mlsanity.com/paper/2409.17126/) (2024)
- [Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Models](https://mlsanity.com/paper/2502.10090/) (2025)
- [Rearrangement Planning for General Part Assembly](https://mlsanity.com/paper/2307.00206/) (2023)
- [SPAFormer: Sequential 3D Part Assembly with Transformers](https://mlsanity.com/paper/2403.05874/) (2024)
- [Manual-PA: Learning 3D Part Assembly from Instruction Diagrams](https://mlsanity.com/paper/2411.18011/) (2024)
- [Toward Automated Programming for Robotic Assembly Using ChatGPT](https://mlsanity.com/paper/2405.08216/) (2024)
- [Component Selection for Craft Assembly Tasks](https://mlsanity.com/paper/2407.14001/) (2024)
- ["Stack It Up!": 3D Stable Structure Generation from 2D Hand-drawn Sketch](https://mlsanity.com/paper/2508.02093/) (2025)

## Abstract (arXiv)

Inspired by traditional handmade crafts, where a person improvises assemblies based on the available objects, we formally introduce the Craft Assembly Task. It is a robotic assembly task that involves building an accurate representation of a given target object using the available objects, which do not directly correspond to its parts. In this work, we focus on selecting the subset of available objects for the final craft, when the given input is an RGB image of the target in the wild. We use a mask segmentation neural network to identify visible parts, followed by retrieving labeled template meshes. These meshes undergo pose optimization to determine the most suitable template. Then, we propose to simplify the parts of the transformed template mesh to primitive shapes like cuboids or cylinders. Finally, we design a search algorithm to find correspondences in the scene based on local and global proportions. We develop baselines for comparison that consider all possible combinations, and choose the highest scoring combination for common metrics used in foreground maps and mask accuracy. Our approach achieves comparable results to the baselines for two different scenes, and we show qualitative results for an implementation in a real-world scenario.