# 3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows

arXiv: https://arxiv.org/abs/2210.11603 · HTML: https://mlsanity.com/paper/2210.11603/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 99 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [ABC: A Big CAD Model Dataset For Geometric Deep Learning](https://mlsanity.com/paper/1812.06216/) (2018)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation](https://mlsanity.com/paper/2110.02624/) (2021)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [MirrorGAN: Learning Text-to-image Generation by Redescription](https://mlsanity.com/paper/1903.05854/) (2019)
- [Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims](https://mlsanity.com/paper/2004.07213/) (2020)
- [PromptChainer: Chaining Large Language Model Prompts through Visual Programming](https://mlsanity.com/paper/2203.06566/) (2022)
- [Sparks: Inspiration for Science Writing using Language Models](https://mlsanity.com/paper/2110.07640/) (2021)
- [Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation](https://mlsanity.com/paper/1909.11081/) (2019)
- [Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequences](https://mlsanity.com/paper/2010.02392/) (2020)

## Built on by (most cited first, 30 of 41)

- [Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models](https://mlsanity.com/paper/2304.09337/) (2023)
- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing](https://mlsanity.com/paper/2402.10294/) (2024)
- [CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Language](https://mlsanity.com/paper/2211.01427/) (2022)
- [ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles](https://mlsanity.com/paper/2310.15428/) (2023)
- [Query2CAD: Generating CAD models using natural language queries](https://mlsanity.com/paper/2406.00144/) (2024)
- [A Comprehensive Survey on 3D Content Generation](https://mlsanity.com/paper/2402.01166/) (2024)
- [Generative Disco: Text-to-Video Generation for Music Visualization](https://mlsanity.com/paper/2304.08551/) (2023)
- [Large Language Models for Computer-Aided Design: A Survey](https://mlsanity.com/paper/2505.08137/) (2025)
- [VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representation](https://mlsanity.com/paper/2307.16605/) (2023)
- [Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI Workflow](https://mlsanity.com/paper/2402.09894/) (2024)
- [Next Steps for Human-Centered Generative AI: A Technical Perspective](https://mlsanity.com/paper/2306.15774/) (2023)
- [CADSmith: Multi-Agent CAD Generation with Programmatic Geometric Validation](https://mlsanity.com/paper/2603.26512/) (2026)
- [DreamGarden: A Designer Assistant for Growing Games from a Single Prompt](https://mlsanity.com/paper/2410.01791/) (2024)
- [Text2Robot: Evolutionary Robot Design from Text Descriptions](https://mlsanity.com/paper/2406.19963/) (2024)
- [PromptMap: An Alternative Interaction Style for AI-Based Image Generation](https://mlsanity.com/paper/2503.09436/) (2025)
- [PromptCrafter: Crafting Text-to-Image Prompt through Mixed-Initiative Dialogue with LLM](https://mlsanity.com/paper/2307.08985/) (2023)
- [Language Models as Critical Thinking Tools: A Case Study of Philosophers](https://mlsanity.com/paper/2404.04516/) (2024)
- [SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketches](https://mlsanity.com/paper/2502.07556/) (2025)
- [DesignWeaver: Dimensional Scaffolding for Text-to-Image Product Design](https://mlsanity.com/paper/2502.09867/) (2025)
- [POET: Supporting Prompting Creativity and Personalization with Automated Expansion of Text-to-Image Generation](https://mlsanity.com/paper/2504.13392/) (2025)
- [GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design](https://mlsanity.com/paper/2508.15227/) (2025)
- [SCAPE: Searching Conceptual Architecture Prompts using Evolution](https://mlsanity.com/paper/2402.00089/) (2024)
- [Survey of User Interface Design and Interaction Techniques in Generative AI Applications](https://mlsanity.com/paper/2410.22370/) (2024)
- [Hookpad Aria: A Copilot for Songwriters](https://mlsanity.com/paper/2502.08122/) (2025)
- [CAD-Prompted Generative Models: A Pathway to Feasible and Novel Engineering Designs](https://mlsanity.com/paper/2407.08675/) (2024)
- [Zero-Shot Subject-Centric Generation for Creative Application Using Entropy Fusion](https://mlsanity.com/paper/2503.10697/) (2025)
- [Enabling Generative Design Tools with LLM Agents for Mechanical Computation Devices: A Case Study](https://mlsanity.com/paper/2405.17837/) (2024)
- [Reimagining Personal Data: Unlocking the Potential of AI-Generated Images in Personal Data Meaning-Making](https://mlsanity.com/paper/2502.18853/) (2025)
- [FeedQUAC: Quick Unobtrusive AI-Generated Commentary](https://mlsanity.com/paper/2504.16416/) (2025)

## Abstract (arXiv)

Text-to-image AI are capable of generating novel images for inspiration, but their applications for 3D design workflows and how designers can build 3D models using AI-provided inspiration have not yet been explored. To investigate this, we integrated DALL-E, GPT-3, and CLIP within a CAD software in 3DALL-E, a plugin that generates 2D image inspiration for 3D design. 3DALL-E allows users to construct text and image prompts based on what they are modeling. In a study with 13 designers, we found that designers saw great potential in 3DALL-E within their workflows and could use text-to-image AI to produce reference images, prevent design fixation, and inspire design considerations. We elaborate on prompting patterns observed across 3D modeling tasks and provide measures of prompt complexity observed across participants. From our findings, we discuss how 3DALL-E can merge with existing generative design workflows and propose prompt bibliographies as a form of human-AI design history.