Data to 7 Oct 2026

Optimizing Prompts for Text-to-Image Generation

arXiv 2212.09611 ↗First posted 19 Dec 2022cs.CL cs.CVcited by 204

Not on the map. Why?

Cited by 88 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Well-designed prompts can guide text-to-image models to generate amazing images. However, the performant prompts are often model-specific and misaligned with user input. Instead of laborious human engineering, we propose prompt adaptation, a general framework that automatically adapts original user input to model-preferred prompts. Specifically, we first perform supervised fine-tuning with a pretrained language model on a small collection of manually engineered prompts. Then we use reinforcement learning to explore better prompts. We define a reward function that encourages the policy to generate more aesthetically pleasing images while preserving the original user intentions. Experimental results on Stable Diffusion show that our method outperforms manual prompt engineering in terms of both automatic metrics and human preference ratings. Moreover, reinforcement learning further boosts performance, especially on out-of-domain prompts. The pretrained checkpoints are available at https://aka.ms/promptist. The demo can be found at https://aka.ms/promptist-demo.

Key facts. Cited by 204 papers in the map (data through 7 Oct 2026). Builds on 45 of its 51 references that are in the map. First posted on arXiv 19 Dec 2022.

Builds on

45 of 51 references.

References · 45

2017Attention Is All You Needcited by 59,932
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2020Language Models are Few-Shot Learnerscited by 32,199
2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
2022Training language models to follow instructions with human feedbackcited by 15,712
2017Proximal Policy Optimization Algorithmscited by 14,441
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2021Zero-Shot Text-to-Image Generationcited by 4,276

33 more

2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
2017Deep reinforcement learning from human preferencescited by 3,846
2020Learning to summarize from human feedbackcited by 2,562
2021Learning to Prompt for Vision-Language Modelscited by 2,269
2019Fine-Tuning Language Models from Human Preferencescited by 2,124
2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
2019Language Models as Knowledge Bases?cited by 1,922
2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
2019How Can We Know What Language Models Know?cited by 953
2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
2016Generative Adversarial Text to Image Synthesiscited by 735
2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
2021Knowledge Neurons in Pretrained Transformerscited by 590
2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
2018Reward learning from human preferences and demonstrations in Ataricited by 314
2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 162
2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
2016Learning What and Where to Drawcited by 105
2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
2021BERTese: Learning to Speak to BERTcited by 82
2022A Taxonomy of Prompt Modifiers for Text-To-Image Generationcited by 54
2019Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Studycited by 40
2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedbackcited by 35
2020Deploying Lifelong Open-Domain Dialogue Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 42; 2024: 53; 2025: 70; 2026: 39.

204 papers.

Sort:

By area:

2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 284
2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 171
2023Holistic Evaluation of Text-To-Image Modelscited by 160
2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
2024Aligning Diffusion Models by Optimizing Human Utilitycited by 99

38 more

2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 91
2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
2023A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesiscited by 78
2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
2023PromptMagician: Interactive Prompt Engineering for Text-to-Image Creationcited by 58
2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
2024UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Imagescited by 52
2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
2023Prompt Stealing Attacks Against Text-to-Image Generation Modelscited by 48
2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generationcited by 47
2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
2024DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusioncited by 46
2024Dynamic Prompt Optimizing for Text-to-Image Generationcited by 45
2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
2025SnapMoGen: Human Motion Generation from Expressive Textscited by 44
2023What's the Magic Word? A Control Theory of LLM Promptingcited by 41
2025PromptEnhancer: A Simple Approach to Enhance Text-to-Image Models via Chain-of-Thought Prompt Rewritingcited by 38
2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
2024Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generationcited by 35
2025Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learningcited by 33
2025T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generationcited by 31
2023Enhancing Diffusion Models with Text-Encoder Reinforcement Learningcited by 30
2024Accelerating Parallel Sampling of Diffusion Modelscited by 30
2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibrationcited by 30
2023AutoHint: Automatic Prompt Optimization with Hint Generationcited by 29
2024Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biasescited by 29
2023RadEdit: stress-testing biomedical vision models via diffusion image editingcited by 28
2024On Discrete Prompt Optimization for Diffusion Modelscited by 28
2024TextCraftor: Your Text Encoder Can be Image Quality Controllercited by 27
2025VPO: Aligning Text-to-Video Generation Models with Prompt Optimizationcited by 27
2023Visual Instruction Inversion: Image Editing via Visual Promptingcited by 26
2023Plan, Posture and Go: Towards Open-World Text-to-Motion Generationcited by 26
2025VISTA: A Test-Time Self-Improving Video Generation Agentcited by 26
2023Improving Compositional Text-to-image Generation with Large Vision-Language Modelscited by 24

50 most cited of 204 All 204 (JSON)

Similar, not cited

Close in wording, no citation either way