# BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesis

arXiv: https://arxiv.org/abs/2311.06752 · HTML: https://mlsanity.com/paper/2311.06752/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 12 Nov 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](https://mlsanity.com/paper/2303.14420/) (2023)
- [A Taxonomy of Prompt Modifiers for Text-To-Image Generation](https://mlsanity.com/paper/2204.13988/) (2022)
- [Best Prompts for Text-to-Image Models and How to Find Them](https://mlsanity.com/paper/2209.11711/) (2022)

## Built on by (most cited first, 18 of 18)

- [RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learning](https://mlsanity.com/paper/2505.17540/) (2025)
- [Maestro: Self-Improving Text-to-Image Generation via Agent Orchestration](https://mlsanity.com/paper/2509.10704/) (2025)
- [RealGen: Photorealistic Text-to-Image Generation via Detector-Guided Rewards](https://mlsanity.com/paper/2512.00473/) (2025)
- [ChatGen: Automatic Text-to-Image Generation From FreeStyle Chatting](https://mlsanity.com/paper/2411.17176/) (2024)
- [Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images](https://mlsanity.com/paper/2404.13784/) (2024)
- [ReNeg: Learning Negative Embedding with Reward Guidance](https://mlsanity.com/paper/2412.19637/) (2024)
- [PromptMap: An Alternative Interaction Style for AI-Based Image Generation](https://mlsanity.com/paper/2503.09436/) (2025)
- [Endogenous Reprompting: Self-Evolving Cognitive Alignment for Unified Multimodal Models](https://mlsanity.com/paper/2601.20305/) (2026)
- [SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketches](https://mlsanity.com/paper/2502.07556/) (2025)
- [From Noise to Nuance: Advances in Deep Generative Image Models](https://mlsanity.com/paper/2412.09656/) (2024)
- [ArtAug: Enhancing Text-to-Image Generation through Synthesis-Understanding Interaction](https://mlsanity.com/paper/2412.12888/) (2024)
- [EPIG: Emotion-Based Prompting for Personalised Image Generation](https://mlsanity.com/paper/2606.13247/) (2026)
- [IQA-Adapter: Exploring Knowledge Transfer from Image Quality Assessment to Diffusion-based Generative Models](https://mlsanity.com/paper/2412.01794/) (2024)
- [AdaptaGen: Domain-Specific Image Generation through Hierarchical Semantic Optimization Framework](https://mlsanity.com/paper/2507.05621/) (2025)
- [Seeing is Believing: Aligning Prompt Rewriting with Visual Anchors for Text-to-Image Generation](https://mlsanity.com/paper/2606.08492/) (2026)
- [EvoGen-Harness: Learning Where and How to Evolve Image-Generation Harnesses](https://mlsanity.com/paper/2610.00383/) (2026)
- [TIDE: Trajectory-Informed Skill Distillation for Efficient Text-to-Video Prompt Optimization](https://mlsanity.com/paper/2610.15098/) (2026)
- [FlowCred: Hierarchical Credit Assignment and Functional Coordination for Joint Text-to-Video Post-Training](https://mlsanity.com/paper/2610.18142/) (2026)

## Abstract (arXiv)

Recently, diffusion-based deep generative models (e.g., Stable Diffusion) have shown impressive results in text-to-image synthesis. However, current text-to-image models often require multiple passes of prompt engineering by humans in order to produce satisfactory results for real-world applications. We propose BeautifulPrompt, a deep generative model to produce high-quality prompts from very simple raw descriptions, which enables diffusion-based models to generate more beautiful images. In our work, we first fine-tuned the BeautifulPrompt model over low-quality and high-quality collecting prompt pairs. Then, to ensure that our generated prompts can generate more beautiful images, we further propose a Reinforcement Learning with Visual AI Feedback technique to fine-tune our model to maximize the reward values of the generated prompts, where the reward values are calculated based on the PickScore and the Aesthetic Scores. Our results demonstrate that learning from visual AI feedback promises the potential to improve the quality of generated prompts and images significantly. We further showcase the integration of BeautifulPrompt to a cloud-native AI platform to provide better text-to-image generation service in the cloud.