# Best Prompts for Text-to-Image Models and How to Find Them

arXiv: https://arxiv.org/abs/2209.11711 · HTML: https://mlsanity.com/paper/2209.11711/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [How Context Affects Language Models' Factual Predictions](https://mlsanity.com/paper/2005.04611/) (2020)
- [Learning from Crowds with Crowd-Kit](https://mlsanity.com/paper/2109.08584/) (2021)
- [Image Composition Assessment with Saliency-augmented Multi-pattern Pooling](https://mlsanity.com/paper/2104.03133/) (2021)
- [IMDB-WIKI-SbS: An Evaluation Dataset for Crowdsourced Pairwise Comparisons](https://mlsanity.com/paper/2110.14990/) (2021)

## Built on by (most cited first, 30 of 35)

- [T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.08453/) (2023)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models](https://mlsanity.com/paper/2305.13873/) (2023)
- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [Prompt Stealing Attacks Against Text-to-Image Generation Models](https://mlsanity.com/paper/2302.09923/) (2023)
- [Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis](https://mlsanity.com/paper/2402.18078/) (2024)
- [Dynamic Prompt Optimizing for Text-to-Image Generation](https://mlsanity.com/paper/2404.04095/) (2024)
- [Diffusion Facial Forgery Detection](https://mlsanity.com/paper/2401.15859/) (2024)
- [A Prompt Log Analysis of Text-to-Image Generation Systems](https://mlsanity.com/paper/2303.04587/) (2023)
- [Q-Refine: A Perceptual Quality Refiner for AI-Generated Image](https://mlsanity.com/paper/2401.01117/) (2024)
- [Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation](https://mlsanity.com/paper/2403.12075/) (2024)
- [Language-Driven Interactive Traffic Trajectory Generation](https://mlsanity.com/paper/2405.15388/) (2024)
- [Text-to-Image Generation for Abstract Concepts](https://mlsanity.com/paper/2309.14623/) (2023)
- [BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesis](https://mlsanity.com/paper/2311.06752/) (2023)
- [Diffusion Explainer: Visual Explanation for Text-to-image Stable Diffusion](https://mlsanity.com/paper/2305.03509/) (2023)
- [Generating Compositional Scenes via Text-to-image RGBA Instance Generation](https://mlsanity.com/paper/2411.10913/) (2024)
- [Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understanding](https://mlsanity.com/paper/2404.11589/) (2024)
- [Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis](https://mlsanity.com/paper/2411.16503/) (2024)
- [Vision + Language Applications: A Survey](https://mlsanity.com/paper/2305.14598/) (2023)
- [PromptMap: An Alternative Interaction Style for AI-Based Image Generation](https://mlsanity.com/paper/2503.09436/) (2025)
- [No Longer Trending on Artstation: Prompt Analysis of Generative AI Art](https://mlsanity.com/paper/2401.14425/) (2024)
- [VisualPrompter: Semantic-Aware Prompt Optimization with Visual Feedback for Text-to-Image Synthesis](https://mlsanity.com/paper/2506.23138/) (2025)
- [SCAPE: Searching Conceptual Architecture Prompts using Evolution](https://mlsanity.com/paper/2402.00089/) (2024)
- [CoProNN: Concept-based Prototypical Nearest Neighbors for Explaining Vision Models](https://mlsanity.com/paper/2404.14830/) (2024)
- [PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction](https://mlsanity.com/paper/2406.04746/) (2024)
- [Grounding Text-to-Image Diffusion Models for Controlled High-Quality Image Generation](https://mlsanity.com/paper/2501.09194/) (2025)
- [Human Aesthetic Preference-Based Large Text-to-Image Model Personalization: Kandinsky Generation as an Example](https://mlsanity.com/paper/2402.06389/) (2024)
- [MineTheGap: Automatic Mining of Biases in Text-to-Image Models](https://mlsanity.com/paper/2512.13427/) (2025)
- [One Rewrite to Fix Them All? Type-Aware Repair Allocation for Text-to-Image Prompt Optimization](https://mlsanity.com/paper/2607.18724/) (2026)
- [Exploring the Effectiveness of Dataset Synthesis: An application of Apple Detection in Orchards](https://mlsanity.com/paper/2306.11763/) (2023)

## Abstract (arXiv)

Recent progress in generative models, especially in text-guided diffusion models, has enabled the production of aesthetically-pleasing imagery resembling the works of professional human artists. However, one has to carefully compose the textual description, called the prompt, and augment it with a set of clarifying keywords. Since aesthetics are challenging to evaluate computationally, human feedback is needed to determine the optimal prompt formulation and keyword combination. In this paper, we present a human-in-the-loop approach to learning the most useful combination of prompt keywords using a genetic algorithm. We also show how such an approach can improve the aesthetic appeal of images depicting the same descriptions.