mlsanityData to 7 Oct 2026

Dynamic Prompt Optimizing for Text-to-Image Generation

arXiv 2404.04095 ↗First posted 5 Apr 2024cs.CV cs.AIcited by 45

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Text-to-image generative models, specifically those based on diffusion models like Imagen and Stable Diffusion, have made substantial advancements. Recently, there has been a surge of interest in the delicate refinement of text prompts. Users assign weights or alter the injection time steps of certain words in the text prompts to improve the quality of generated images. However, the success of fine-control prompts depends on the accuracy of the text prompts and the careful selection of weights and time steps, which requires significant manual intervention. To address this, we introduce the Prompt Auto-Editing (PAE) method. Besides refining the original prompts for image generation, we further employ an online reinforcement learning strategy to explore the weights and injection time steps of each word, leading to the dynamic fine-control prompts. The reward function during training encourages the model to consider aesthetic score, semantic consistency, and user preferences. Experimental results demonstrate that our proposed method effectively improves the original prompts, generating visually more appealing images while maintaining semantic alignment. Code is available at https://github.com/Mowenyii/PAE.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 34 of its 42 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

34 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Model15 Nov
  2. 2022Best Prompts for Text-to-Image Models and How to Find Them23 Sep
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2017Proximal Policy Optimization Algorithmscited by 14,441
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2022Scalable Diffusion Models with Transformerscited by 5,931
  12. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
17 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  5. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  6. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  7. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 441
  8. 2023Rethinking FID: Towards a Better Evaluation Metric for Image Generationcited by 303
  9. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  10. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  11. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  12. 2022The Creativity of Text-to-Image Generationcited by 73
  13. 2022A Taxonomy of Prompt Modifiers for Text-To-Image Generationcited by 54
  14. 2023A Prompt Log Analysis of Text-to-Image Generation Systemscited by 24
  15. 2023ArtWhisperer: A Dataset for Characterizing Human-AI Interactions in Artistic Creationscited by 10
  16. 2023Collaborative Generative AI: Integrating GPT-k for Efficient Editing in Text-to-Image Generationcited by 4
  17. 2023Intriguing Property and Counterfactual Explanation of GAN for Remote Sensing Image Generationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 25; 2026: 13.

45 papers.

Sort:

By area:

  1. 2024Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLMcited by 22
  2. 2025The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generationcited by 21
  3. 2025Maestro: Self-Improving Text-to-Image Generation via Agent Orchestrationcited by 18
  4. 2026Think-Then-Generate: Reasoning-Aware Text-to-Image Diffusion with LLM Encoderscited by 15
  5. 2025Enhancing Reward Models for High-quality Image Generation: Beyond Text-Image Alignmentcited by 14
  6. 2025ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generationcited by 14
  7. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 11
  8. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 11
  9. 2026PromptRL: Prompt Matters in RL for Flow-Based Image Generationcited by 11
  10. 2025RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scalingcited by 9
  11. 2024CoCoNO: Attention Contrast-and-Complete for Initial Noise Optimization in Text-to-Image Synthesiscited by 7
  12. 2024Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Surveycited by 6
33 more
  1. 2024TIPO: Text to Image with Text Presampling for Prompt Optimizationcited by 6
  2. 2025Regulatory DNA sequence Design with Reinforcement Learningcited by 6
  3. 2026SOAR: Self-Correction for Optimal Alignment and Refinement in Diffusion Modelscited by 6
  4. 2025Causally Steered Diffusion for Automated Video Counterfactual Generationcited by 5
  5. 2025Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Modelscited by 5
  6. 2025Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMscited by 5
  7. 2025Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Modelscited by 5
  8. 2024Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspectivecited by 4
  9. 2025World-To-Image: Grounding Text-to-Image Generation with Agent-Driven World Knowledgecited by 4
  10. 2025Guiding What Not to Generate: Automated Negative Prompting for Text-Image Alignmentcited by 4
  11. 2026AccelAes: Accelerating Diffusion Transformers for Training-Free Aesthetic-Enhanced Image Generationcited by 4
  12. 2024Uniform Attention Maps: Boosting Image Fidelity in Reconstruction and Editingcited by 2
  13. 2025PointT2I: LLM-based text-to-image generation via keypointscited by 2
  14. 2025AutoEdit: Automatic Hyperparameter Tuning for Image Editingcited by 2
  15. 2026Premier: Personalized Preference Modulation with Learnable User Embedding in Text-to-Image Generationcited by 2
  16. 2025InstructEngine: Instruction-driven Text-to-Image Alignmentcited by 1
  17. 2025A Data-Centric Approach to Pedestrian Attribute Recognition: Synthetic Augmentation via Prompt-driven Diffusion Modelscited by 1
  18. 2025MetaLogic: Robustness Evaluation of Text-to-Image Models via Logically Equivalent Promptscited by 1
  19. 2026IdGlow: Dynamic Identity Modulation for Multi-Subject Generationcited by 1
  20. 2026Banana100: Breaking NR-IQA Metrics by 100 Iterative Image Replications with Nano Banana Procited by 1
  21. 2026PromptEvolver: Prompt Inversion through Evolutionary Optimization in Natural-Language Spacecited by 1
  22. 2026Efficient Multi-objective Prompt Optimization via Pure-exploration Banditscited by 1
  23. 2025PromptIQ: Who Cares About Prompts? Let System Handle It -- A Component-Aware Framework for T2I Generationcited by 0
  24. 2025Rhetorical Text-to-Image Generation via Two-layer Diffusion Policy Optimizationcited by 0
  25. 2025GenEscape: Hierarchical Multi-Agent Generation of Escape Room Puzzlescited by 0
  26. 2025Mask Consistency Regularization in Object Removalcited by 0
  27. 2025A Reason-then-Describe Instruction Interpreter for Controllable Video Generationcited by 0
  28. 20252K-Characters-10K-Stories: A Quality-Gated Stylized Narrative Dataset with Disentangled Control and Sequence Consistencycited by 0
  29. 2025Exposing Hidden Biases in Text-to-Image Models via Automated Prompt Searchcited by 0
  30. 2026P-Flow: Prompting Visual Effects Generationcited by 0
  31. 2026ExpressEdit: Fast Editing of Stylized Facial Expressions with Diffusion Models in Photoshopcited by 0
  32. 2026Attention Dynamics in Diffusion Models: A Visual Analytics Framework for Human-AI Collaborationcited by 0
  33. 2026DOGS: Design-Space Sampling for Prompt-Driven Logo Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning to Sample Effective and Diverse Prompts for Text-to-Image Generation
  2. 2024Batch-Instructed Gradient for Prompt Evolution:Systematic Prompt Optimization for Enhanced Text-to-Image Synthesis
  3. 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
  4. 2025PromptEnhancer: A Simple Approach to Enhance Text-to-Image Models via Chain-of-Thought Prompt Rewriting
  5. 2024Powerful and Flexible: Personalized Text-to-Image Generation via Reinforcement Learning
  6. 2023Dynamic Prompt Learning: Addressing Cross-Attention Leakage for Text-Based Image Editing
  7. 2023BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesis
  8. 2025POET: Preference Optimization for Enhanced Text-to-Image Generation