# A Taxonomy of Prompt Modifiers for Text-To-Image Generation

arXiv: https://arxiv.org/abs/2204.13988 · HTML: https://mlsanity.com/paper/2204.13988/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 72 references that are in the map.
- First posted on arXiv 20 Apr 2022.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [Cooperating with Machines](https://mlsanity.com/paper/1703.06207/) (2017)
- [Autonomy, Authenticity, Authorship and Intention in computer generated art](https://mlsanity.com/paper/1903.02166/) (2019)
- [Declarative Machine Learning Systems](https://mlsanity.com/paper/2107.08148/) (2021)

## Built on by (most cited first, 30 of 55)

- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [Optimizing Prompts for Text-to-Image Generation](https://mlsanity.com/paper/2212.09611/) (2022)
- [Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models](https://mlsanity.com/paper/2305.13873/) (2023)
- [Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator](https://mlsanity.com/paper/2309.14494/) (2023)
- [Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models](https://mlsanity.com/paper/2304.09337/) (2023)
- [PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation](https://mlsanity.com/paper/2307.09036/) (2023)
- [Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?](https://mlsanity.com/paper/2302.03495/) (2023)
- [Prompt Stealing Attacks Against Text-to-Image Generation Models](https://mlsanity.com/paper/2302.09923/) (2023)
- [Dynamic Prompt Optimizing for Text-to-Image Generation](https://mlsanity.com/paper/2404.04095/) (2024)
- [Zero-shot Generation of Coherent Storybook from Plain Text Story using Diffusion Models](https://mlsanity.com/paper/2302.03900/) (2023)
- [Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLM](https://mlsanity.com/paper/2412.15156/) (2024)
- [A Prompt Log Analysis of Text-to-Image Generation Systems](https://mlsanity.com/paper/2303.04587/) (2023)
- [Large Language Model Agent as a Mechanical Designer](https://mlsanity.com/paper/2404.17525/) (2024)
- [TIAM -- A Metric for Evaluating Alignment in Text-to-Image Generation](https://mlsanity.com/paper/2307.05134/) (2023)
- [Text-to-Image Generation for Abstract Concepts](https://mlsanity.com/paper/2309.14623/) (2023)
- [BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesis](https://mlsanity.com/paper/2311.06752/) (2023)
- [Diffusion Explainer: Visual Explanation for Text-to-image Stable Diffusion](https://mlsanity.com/paper/2305.03509/) (2023)
- [Adjacent Words, Divergent Intents: Jailbreaking Large Language Models via Task Concurrency](https://mlsanity.com/paper/2510.21189/) (2025)
- [Using Text-to-Image Generation for Architectural Design Ideation](https://mlsanity.com/paper/2304.10182/) (2023)
- [RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scaling](https://mlsanity.com/paper/2510.20206/) (2025)
- [Prompt Evolution for Generative AI: A Classifier-Guided Approach](https://mlsanity.com/paper/2305.16347/) (2023)
- [Wordflow: Social Prompt Engineering for Large Language Models](https://mlsanity.com/paper/2401.14447/) (2024)
- [Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images](https://mlsanity.com/paper/2404.13784/) (2024)
- [Vision + Language Applications: A Survey](https://mlsanity.com/paper/2305.14598/) (2023)
- [Learning Disentangled Prompts for Compositional Image Synthesis](https://mlsanity.com/paper/2306.00763/) (2023)
- [Perceptions and Realities of Text-to-Image Generation](https://mlsanity.com/paper/2306.08363/) (2023)
- [Exploring Prompt Engineering: A Systematic Review with SWOT Analysis](https://mlsanity.com/paper/2410.12843/) (2024)
- [Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2506.16853/) (2025)
- [Grimm in Wonderland: Prompt Engineering with Midjourney to Illustrate Fairytales](https://mlsanity.com/paper/2302.08961/) (2023)
- [SCAPE: Searching Conceptual Architecture Prompts using Evolution](https://mlsanity.com/paper/2402.00089/) (2024)

## Abstract (arXiv)

Text-to-image generation has seen an explosion of interest since 2021. Today, beautiful and intriguing digital images and artworks can be synthesized from textual inputs ("prompts") with deep generative models. Online communities around text-to-image generation and AI generated art have quickly emerged. This paper identifies six types of prompt modifiers used by practitioners in the online community based on a 3-month ethnographic study. The novel taxonomy of prompt modifiers provides researchers a conceptual starting point for investigating the practice of text-to-image generation, but may also help practitioners of AI generated art improve their images. We further outline how prompt modifiers are applied in the practice of "prompt engineering." We discuss research opportunities of this novel creative practice in the field of Human-Computer Interaction (HCI). The paper concludes with a discussion of broader implications of prompt engineering from the perspective of Human-AI Interaction (HAI) in future applications beyond the use case of text-to-image generation and AI generated art.