# PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation

arXiv: https://arxiv.org/abs/2307.09036 · HTML: https://mlsanity.com/paper/2307.09036/

## Key facts

- Cited by 58 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 84 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 30 of 53)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold](https://mlsanity.com/paper/2305.10973/) (2023)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [Optimizing Prompts for Text-to-Image Generation](https://mlsanity.com/paper/2212.09611/) (2022)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [Learning a No-Reference Quality Metric for Single-Image Super-Resolution](https://mlsanity.com/paper/1612.05890/) (2016)
- [PromptChainer: Chaining Large Language Model Prompts through Visual Programming](https://mlsanity.com/paper/2203.06566/) (2022)
- [ArtEmis: Affective Language for Visual Art](https://mlsanity.com/paper/2101.07396/) (2021)

## Built on by (most cited first, 30 of 58)

- [PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement](https://mlsanity.com/paper/2403.04014/) (2024)
- [Foundation Models Meet Visualizations: Challenges and Opportunities](https://mlsanity.com/paper/2310.05771/) (2023)
- [MagicQuill: An Intelligent Interactive Image Editing System](https://mlsanity.com/paper/2411.09703/) (2024)
- [AgentCoord: Visually Exploring Coordination Strategy for LLM-based Multi-Agent Collaboration](https://mlsanity.com/paper/2404.11943/) (2024)
- [CharacterFactory: Sampling Consistent Characters with GANs for Diffusion Models](https://mlsanity.com/paper/2404.15677/) (2024)
- [AgentLens: Visual Analysis for Agent Behaviors in LLM-based Autonomous Systems](https://mlsanity.com/paper/2402.08995/) (2024)
- [PromptSculptor: Multi-Agent Based Text-to-Image Prompt Optimization](https://mlsanity.com/paper/2509.12446/) (2025)
- [A Survey on Quality Metrics for Text-to-Image Generation](https://mlsanity.com/paper/2403.11821/) (2024)
- [IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation](https://mlsanity.com/paper/2601.04498/) (2026)
- [Prompt Refinement with Image Pivot for Text-to-Image Generation](https://mlsanity.com/paper/2407.00247/) (2024)
- [Human-Guided Image Generation for Expanding Small-Scale Training Image Datasets](https://mlsanity.com/paper/2412.16839/) (2024)
- [PromptMap: An Alternative Interaction Style for AI-Based Image Generation](https://mlsanity.com/paper/2503.09436/) (2025)
- [DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoning](https://mlsanity.com/paper/2508.01700/) (2025)
- [VideoPro: A Visual Analytics Approach for Interactive Video Programming](https://mlsanity.com/paper/2308.00401/) (2023)
- [JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models](https://mlsanity.com/paper/2404.08793/) (2024)
- [ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map](https://mlsanity.com/paper/2407.12315/) (2024)
- [Prompting for products: Investigating design space exploration strategies for text-to-image generative models](https://mlsanity.com/paper/2408.03946/) (2024)
- [Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directions](https://mlsanity.com/paper/2409.19747/) (2024)
- [Exploring Empty Spaces: Human-in-the-Loop Data Augmentation](https://mlsanity.com/paper/2410.01088/) (2024)
- [SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketches](https://mlsanity.com/paper/2502.07556/) (2025)
- [VisualPrompter: Semantic-Aware Prompt Optimization with Visual Feedback for Text-to-Image Synthesis](https://mlsanity.com/paper/2506.23138/) (2025)
- [ASAP: Interpretable Analysis and Summarization of AI-generated Image Patterns at Scale](https://mlsanity.com/paper/2404.02990/) (2024)
- [Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLM](https://mlsanity.com/paper/2410.14879/) (2024)
- [Exploring Multimodal Prompt for Visualization Authoring with Large Language Models](https://mlsanity.com/paper/2504.13700/) (2025)
- [GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design](https://mlsanity.com/paper/2508.15227/) (2025)
- [World-To-Image: Grounding Text-to-Image Generation with Agent-Driven World Knowledge](https://mlsanity.com/paper/2510.04201/) (2025)
- [From Words to Widgets for Controllable LLM Generation](https://mlsanity.com/paper/2604.10925/) (2026)
- [Visual Analytics for Efficient Image Exploration and User-Guided Image Captioning](https://mlsanity.com/paper/2311.01016/) (2023)
- [Taming Text-to-Image Synthesis for Novices: User-centric Prompt Generation via Multi-turn Guidance](https://mlsanity.com/paper/2408.12910/) (2024)
- [POET: Preference Optimization for Enhanced Text-to-Image Generation](https://mlsanity.com/paper/2510.12041/) (2025)

## Abstract (arXiv)

Generative text-to-image models have gained great popularity among the public for their powerful capability to generate high-quality images based on natural language prompts. However, developing effective prompts for desired images can be challenging due to the complexity and ambiguity of natural language. This research proposes PromptMagician, a visual analysis system that helps users explore the image results and refine the input prompts. The backbone of our system is a prompt recommendation model that takes user prompts as input, retrieves similar prompt-image pairs from DiffusionDB, and identifies special (important and relevant) prompt keywords. To facilitate interactive prompt refinement, PromptMagician introduces a multi-level visualization for the cross-modal embedding of the retrieved images and recommended keywords, and supports users in specifying multiple criteria for personalized exploration. Two usage scenarios, a user study, and expert interviews demonstrate the effectiveness and usability of our system, suggesting it facilitates prompt engineering and improves the creativity support of the generative text-to-image model.