Data to 7 Oct 2026

Design Guidelines for Prompt Engineering Text-to-Image Generative Models

arXiv 2109.06977 ↗First posted 14 Sep 2021cs.HCcited by 216

Not on the map. Why?

Cited by 56 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Text-to-image generative models are a new and powerful way to generate visual artwork. However, the open-ended nature of text as interaction is double-edged; while users can input anything and have access to an infinite range of generations, they also must engage in brute-force trial and error with the text prompt when the result quality is poor. We conduct a study exploring what prompt keywords and model hyperparameters can help produce coherent outputs. In particular, we study prompts structured to include subject and style keywords and investigate success and failure modes of these prompts. Our evaluation of 5493 generations over the course of five experiments spans 51 abstract and concrete subjects as well as 51 abstract and figurative styles. From this evaluation, we present design guidelines that can help people produce better outcomes from text-to-image generative models.

Key facts. Cited by 216 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Sep 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 78; 2024: 55; 2025: 49; 2026: 25.

216 papers.

Sort:

By area:

2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
2022Optimizing Prompts for Text-to-Image Generationcited by 204
2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 171
2023HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generationcited by 104
2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
2023Art and the science of generative AI: A deeper divecited by 93
2023Getting pwn'd by AI: Penetration Testing with Large Language Modelscited by 82
2023Generating images of rare concepts using pre-trained diffusion modelscited by 78

38 more

2023A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesiscited by 78
2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
2023FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasetscited by 68
2023Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Modelscited by 67
2023Graphologue: Exploring Large Language Model Responses with Interactive Diagramscited by 65
2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinementcited by 61
2023PromptMagician: Interactive Prompt Engineering for Text-to-Image Creationcited by 58
2024Design Principles for Generative AI Applicationscited by 57
2022A Taxonomy of Prompt Modifiers for Text-To-Image Generationcited by 54
2023Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?cited by 54
2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Modelscited by 54
2023Prompt Engineering for Healthcare: Methodologies and Applicationscited by 52
2023Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Datacited by 48
2023Prompt Stealing Attacks Against Text-to-Image Generation Modelscited by 48
2024Designing Heterogeneous LLM Agents for Financial Sentiment Analysiscited by 48
2024Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesiscited by 48
2024Dynamic Prompt Optimizing for Text-to-Image Generationcited by 45
2023Enhancing Activity Prediction Models in Drug Discovery with the Ability to Understand Human Languagecited by 44
2023RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressionscited by 43
2023Norm-guided latent space exploration for text-to-image generationcited by 43
2024HanDiffuser: Text-to-Image Generation With Realistic Hand Appearancescited by 42
20223DALL-E: Integrating Text-to-Image AI in 3D Design Workflowscited by 41
2023Multimodal Procedural Planning via Dual Text-Image Promptingcited by 40
2023SentimentGPT: Exploiting GPT for Advanced Sentiment Analysis and its Departure from Current Machine Learningcited by 38
2022Best Prompts for Text-to-Image Models and How to Find Themcited by 35
2023PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactionscited by 34
2023CoQuest: Exploring Research Question Co-Creation with an LLM-based Agentcited by 34
2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
2023I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphorscited by 31
2023Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Promptscited by 30
2023AutoVP: An Automated Visual Prompting Framework and Benchmarkcited by 29
2024An Early Categorization of Prompt Injection Attacks on Large Language Modelscited by 28
2024On Discrete Prompt Optimization for Diffusion Modelscited by 28
2023Zero-shot Generation of Coherent Storybook from Plain Text Story using Diffusion Modelscited by 26
2024Supporting Sensemaking of Large Language Model Outputs at Scalecited by 26
2023A Prompt Log Analysis of Text-to-Image Generation Systemscited by 24

50 most cited of 216 All 216 (JSON)