# Sketch2Prototype: Rapid Conceptual Design Exploration and Prototyping with Generative AI

arXiv: https://arxiv.org/abs/2405.12985 · HTML: https://mlsanity.com/paper/2405.12985/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation](https://mlsanity.com/paper/2309.16653/) (2023)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [Shap-E: Generating Conditional 3D Implicit Functions](https://mlsanity.com/paper/2305.02463/) (2023)
- [RealFusion: 360deg Reconstruction of Any Object from a Single Image](https://mlsanity.com/paper/2302.10663/) (2023)
- [From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design](https://mlsanity.com/paper/2311.12668/) (2023)

## Built on by (most cited first, 13 of 13)

- [Query2CAD: Generating CAD models using natural language queries](https://mlsanity.com/paper/2406.00144/) (2024)
- [AI Agents in Engineering Design: A Multi-Agent Framework for Aesthetic and Aerodynamic Car Design](https://mlsanity.com/paper/2503.23315/) (2025)
- [CADSmith: Multi-Agent CAD Generation with Programmatic Geometric Validation](https://mlsanity.com/paper/2603.26512/) (2026)
- [Speech to Reality: On-Demand Production using Natural Language, 3D Generative AI, and Discrete Robotic Assembly](https://mlsanity.com/paper/2409.18390/) (2024)
- [Reimagining Personal Data: Unlocking the Potential of AI-Generated Images in Personal Data Meaning-Making](https://mlsanity.com/paper/2502.18853/) (2025)
- [Toward Knowledge-Guided AI for Inverse Design in Manufacturing: A Perspective on Domain, Physics, and Human-AI Synergy](https://mlsanity.com/paper/2506.00056/) (2025)
- [Text to Robotic Assembly of Multi Component Objects using 3D Generative AI and Vision Language Models](https://mlsanity.com/paper/2511.02162/) (2025)
- [Evaluation of Architectural Synthesis Using Generative AI](https://mlsanity.com/paper/2503.02861/) (2025)
- [Investigating Social Bias in Narrative Image Generation](https://mlsanity.com/paper/2608.01780/) (2026)
- [Creativity in the Age of AI: Evaluating the Impact of Generative AI on Design Outputs and Designers' Creative Thinking](https://mlsanity.com/paper/2411.00168/) (2024)
- [PartStickers: Generating Parts of Objects for Rapid Prototyping](https://mlsanity.com/paper/2504.05508/) (2025)
- [Agentic AI in Engineering and Manufacturing: Industry Perspectives on Utility, Adoption, Challenges, and Opportunities](https://mlsanity.com/paper/2604.09633/) (2026)
- [SemanticSlider3D: Training-Free Continuous Semantic Editing for 3D Objects](https://mlsanity.com/paper/2608.18560/) (2026)

## Abstract (arXiv)

Sketch2Prototype is an AI-based framework that transforms a hand-drawn sketch into a diverse set of 2D images and 3D prototypes through sketch-to-text, text-to-image, and image-to-3D stages. This framework, shown across various sketches, rapidly generates text, image, and 3D modalities for enhanced early-stage design exploration. We show that using text as an intermediate modality outperforms direct sketch-to-3D baselines for generating diverse and manufacturable 3D models. We find limitations in current image-to-3D techniques, while noting the value of the text modality for user-feedback and iterative design augmentation.