mlsanityData to 14 Oct 2026

A Taxonomy of Prompt Modifiers for Text-To-Image Generation

arXiv 2204.13988 ↗First posted 20 Apr 2022cs.MM cs.CL cs.HCcited by 55

Not on the map. Why?

Abstract (arXiv)

Text-to-image generation has seen an explosion of interest since 2021. Today, beautiful and intriguing digital images and artworks can be synthesized from textual inputs ("prompts") with deep generative models. Online communities around text-to-image generation and AI generated art have quickly emerged. This paper identifies six types of prompt modifiers used by practitioners in the online community based on a 3-month ethnographic study. The novel taxonomy of prompt modifiers provides researchers a conceptual starting point for investigating the practice of text-to-image generation, but may also help practitioners of AI generated art improve their images. We further outline how prompt modifiers are applied in the practice of "prompt engineering." We discuss research opportunities of this novel creative practice in the field of Human-Computer Interaction (HCI). The paper concludes with a discussion of broader implications of prompt engineering from the perspective of Human-AI Interaction (HAI) in future applications beyond the use case of text-to-image generation and AI generated art.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Builds on 16 of its 72 references that are in the map. First posted on arXiv 20 Apr 2022.

Builds on

16 of 72 references.

References · 16

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  9. 2020Artificial Intelligence, Values and Alignmentcited by 448
  10. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  11. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  12. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 218
4 more
  1. 2020Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthycited by 183
  2. 2017Cooperating with Machinescited by 39
  3. 2019Autonomy, Authenticity, Authorship and Intention in computer generated artcited by 18
  4. 2021Declarative Machine Learning Systemscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 22; 2024: 12; 2025: 13; 2026: 6.

55 papers.

Sort:

By area:

  1. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 379
  2. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  3. 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 175
  4. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  5. 2023Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Modelscited by 67
  6. 2023PromptMagician: Interactive Prompt Engineering for Text-to-Image Creationcited by 58
  7. 2023Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?cited by 54
  8. 2023Prompt Stealing Attacks Against Text-to-Image Generation Modelscited by 51
  9. 2024Dynamic Prompt Optimizing for Text-to-Image Generationcited by 45
  10. 2023Zero-shot Generation of Coherent Storybook from Plain Text Story using Diffusion Modelscited by 26
  11. 2024Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLMcited by 25
  12. 2023A Prompt Log Analysis of Text-to-Image Generation Systemscited by 24
38 more
  1. 2024Large Language Model Agent as a Mechanical Designercited by 21
  2. 2023TIAM -- A Metric for Evaluating Alignment in Text-to-Image Generationcited by 18
  3. 2023Text-to-Image Generation for Abstract Conceptscited by 18
  4. 2023BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesiscited by 18
  5. 2023Diffusion Explainer: Visual Explanation for Text-to-image Stable Diffusioncited by 17
  6. 2025Adjacent Words, Divergent Intents: Jailbreaking Large Language Models via Task Concurrencycited by 17
  7. 2023Using Text-to-Image Generation for Architectural Design Ideationcited by 16
  8. 2025RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scalingcited by 10
  9. 2023Prompt Evolution for Generative AI: A Classifier-Guided Approachcited by 9
  10. 2024Wordflow: Social Prompt Engineering for Large Language Modelscited by 9
  11. 2024Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Imagescited by 9
  12. 2023Vision + Language Applications: A Surveycited by 8
  13. 2023Learning Disentangled Prompts for Compositional Image Synthesiscited by 8
  14. 2023Perceptions and Realities of Text-to-Image Generationcited by 5
  15. 2024Exploring Prompt Engineering: A Systematic Review with SWOT Analysiscited by 5
  16. 2025Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Modelscited by 5
  17. 2023Grimm in Wonderland: Prompt Engineering with Midjourney to Illustrate Fairytalescited by 4
  18. 2024SCAPE: Searching Conceptual Architecture Prompts using Evolutioncited by 4
  19. 2024DiffChat: Learning to Chat with Text-to-Image Synthesis Models for Interactive Image Creationcited by 4
  20. 2023IteraTTA: An interface for exploring both text prompts and audio priors in generating music with text-to-audio modelscited by 3
  21. 2025Prompt Engineer: Analyzing Hard and Soft Skill Requirements in the AI Job Marketcited by 3
  22. 2023Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT for Mining Insights at Scalecited by 2
  23. 2024CoProNN: Concept-based Prototypical Nearest Neighbors for Explaining Vision Modelscited by 2
  24. 2025The Cow of Rembrandt - Analyzing Artistic Prompt Interpretation in Text-to-Image Modelscited by 2
  25. 2025Towards Effective Prompt Stealing Attack against Text-to-Image Diffusion Modelscited by 2
  26. 2023Detecting Spurious Correlations via Robust Visual Concepts in Real and AI-Generated Image Classificationcited by 1
  27. 2024PhDGPT: Introducing a psychometric and linguistic dataset about how large language models perceive graduate students and professors in psychologycited by 1
  28. 2025Foundations of GenIRcited by 1
  29. 2025Can Large Language Models Unveil the Mysteries? An Exploration of Their Ability to Unlock Information in Complex Scenarioscited by 1
  30. 2025Diversifying Long Prompt Image Generation through Structured Prompt Embedding Space Samplingcited by 1
  31. 2026One Rewrite to Fix Them All? Type-Aware Repair Allocation for Text-to-Image Prompt Optimizationcited by 1
  32. 2026Position: Natural Language Should Not Fully Replace Formal Languagescited by 1
  33. 2024SSP: A Simple and Safe automatic Prompt engineering method towards realistic image synthesis on LVMcited by 0
  34. 2023Semantic Draw Engineering for Text-to-Image Creationcited by 0
  35. 2024Towards Automated Solution Recipe Generation for Industrial Asset Management with LLMcited by 0
  36. 2025CLEAR: Cue Learning using Evolution for Accurate Recognition Applied to Sustainability Data Extractioncited by 0
  37. 2025IPGO: Indirect Prompt Gradient Optimization for Parameter-Efficient Prompt-level Fine-Tuning on Text-to-Image Modelscited by 0
  38. 2025LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documentscited by 0

50 most cited of 55 All 55 (JSON)