mlsanityData to 7 Oct 2026

3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows

arXiv 2210.11603 ↗First posted 20 Oct 2022cs.HC cs.AI cs.CYcited by 41

Not on the map. Why?

Abstract (arXiv)

Text-to-image AI are capable of generating novel images for inspiration, but their applications for 3D design workflows and how designers can build 3D models using AI-provided inspiration have not yet been explored. To investigate this, we integrated DALL-E, GPT-3, and CLIP within a CAD software in 3DALL-E, a plugin that generates 2D image inspiration for 3D design. 3DALL-E allows users to construct text and image prompts based on what they are modeling. In a study with 13 designers, we found that designers saw great potential in 3DALL-E within their workflows and could use text-to-image AI to produce reference images, prevent design fixation, and inspire design considerations. We elaborate on prompting patterns observed across 3D modeling tasks and provide measures of prompt complexity observed across participants. From our findings, we discuss how 3DALL-E can merge with existing generative design workflows and propose prompt bibliographies as a form of human-AI design history.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 36 of its 99 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

36 of 99 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
24 more
  1. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  2. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  3. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  4. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  5. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  6. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  7. 2018ABC: A Big CAD Model Dataset For Geometric Deep Learningcited by 339
  8. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  9. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  10. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  11. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
  12. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  13. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  14. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 163
  15. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
  16. 2021Sparks: Inspiration for Science Writing using Language Modelscited by 80
  17. 2019Interactive Sketch & Fill: Multiclass Sketch-to-Image Translationcited by 51
  18. 2020Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequencescited by 36
  19. 2022Opal: Multimodal Image Generation for News Illustrationcited by 36
  20. 2017SceneSeer: 3D Scene Design with Natural Languagecited by 20
  21. 2018ChatPainter: Improving Text to Image Generation using Dialoguecited by 18
  22. 2022GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Informationcited by 13
  23. 2022Explainable Computational Creativitycited by 8
  24. 2021Visual Conceptual Blending with Large-scale Language and Vision Modelscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 6; 2024: 14; 2025: 12; 2026: 8.

41 papers.

Sort:

By area:

  1. 2023Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Modelscited by 67
  2. 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinementcited by 61
  3. 2024LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editingcited by 47
  4. 2022CLIP-Sculptor: Zero-Shot Generation of High-Fidelity and Diverse Shapes from Natural Languagecited by 46
  5. 2023ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principlescited by 33
  6. 2024A Comprehensive Survey on 3D Content Generationcited by 28
  7. 2024Query2CAD: Generating CAD models using natural language queriescited by 28
  8. 2023Generative Disco: Text-to-Video Generation for Music Visualizationcited by 19
  9. 2025Large Language Models for Computer-Aided Design: A Surveycited by 16
  10. 2023VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representationcited by 14
  11. 2023Next Steps for Human-Centered Generative AI: A Technical Perspectivecited by 13
  12. 2024Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI Workflowcited by 13
29 more
  1. 2026CADSmith: Multi-Agent CAD Generation with Programmatic Geometric Validationcited by 11
  2. 2024DreamGarden: A Designer Assistant for Growing Games from a Single Promptcited by 10
  3. 2024Text2Robot: Evolutionary Robot Design from Text Descriptionscited by 8
  4. 2025PromptMap: An Alternative Interaction Style for AI-Based Image Generationcited by 8
  5. 2023PromptCrafter: Crafting Text-to-Image Prompt through Mixed-Initiative Dialogue with LLMcited by 6
  6. 2024Language Models as Critical Thinking Tools: A Case Study of Philosopherscited by 6
  7. 2025SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketchescited by 6
  8. 2025DesignWeaver: Dimensional Scaffolding for Text-to-Image Product Designcited by 6
  9. 2025POET: Supporting Prompting Creativity and Personalization with Automated Expansion of Text-to-Image Generationcited by 6
  10. 2025GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Designcited by 5
  11. 2024SCAPE: Searching Conceptual Architecture Prompts using Evolutioncited by 4
  12. 2025Hookpad Aria: A Copilot for Songwriterscited by 4
  13. 2024CAD-Prompted Generative Models: A Pathway to Feasible and Novel Engineering Designscited by 3
  14. 2024Survey of User Interface Design and Interaction Techniques in Generative AI Applicationscited by 3
  15. 2025Zero-Shot Subject-Centric Generation for Creative Application Using Entropy Fusioncited by 3
  16. 2024Enabling Generative Design Tools with LLM Agents for Mechanical Computation Devices: A Case Studycited by 2
  17. 2025Reimagining Personal Data: Unlocking the Potential of AI-Generated Images in Personal Data Meaning-Makingcited by 2
  18. 2025FeedQUAC: Quick Unobtrusive AI-Generated Commentarycited by 2
  19. 2025StoryEnsemble: Enabling Dynamic Exploration & Iteration in the Design Process with AI and Forward-Backward Propagationcited by 2
  20. 2024Data Extrapolation for Text-to-image Generation on Small Datasetscited by 1
  21. 2024Thing2Reality: Transforming 2D Content into Conditioned Multiviews and 3D Gaussian Objects for XR Communicationcited by 0
  22. 2025GraphBrep: Learning B-Rep in Graph Structure for Efficient CAD Generationcited by 0
  23. 2026Reality Copilot: Voice-First Human-AI Collaboration in Mixed Reality Using Large Multimodal Modelscited by 0
  24. 2026From Prompts to Worlds: How Users Iterate, Explore, and Make Sense of AI-Generated 3D Environmentscited by 0
  25. 2026"I Just Need GPT to Refine My Prompts": Rethinking Onboarding and Help-Seeking with Generative 3D Modeling Toolscited by 0
  26. 2026Compos3D: Interactive Part-Based Composition for Creative Control in Generative 3D Modelscited by 0
  27. 2026SemanticSlider3D: Training-Free Continuous Semantic Editing for 3D Objectscited by 0
  28. 2026Do Vision-Language Models Agree on the Affective Qualities of Shape? A Cross-Model Audit for Generative Design Interfacescited by 0
  29. 2026InstructMesh: Selective Refinement of Generative 3D Models for Fabricationcited by 0