mlsanityData to 7 Oct 2026

Improving Text-to-Image Consistency via Automatic Prompt Optimization

arXiv 2403.17804 ↗First posted 26 Mar 2024cs.CV cs.CLcited by 88

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Impressive advances in text-to-image (T2I) generative models have yielded a plethora of high performing models which are able to generate aesthetically appealing, photorealistic images. Despite the progress, these models still struggle to produce images that are consistent with the input prompt, oftentimes failing to capture object quantities, relations and attributes properly. Existing solutions to improve prompt-image consistency suffer from the following challenges: (1) they oftentimes require model fine-tuning, (2) they only focus on nearby prompt samples, and (3) they are affected by unfavorable trade-offs among image quality, representation diversity, and prompt-image consistency. In this paper, we address these challenges and introduce a T2I optimization-by-prompting framework, OPT2I, which leverages a large language model (LLM) to improve prompt-image consistency in T2I models. Our framework starts from a user prompt and iteratively generates revised prompts with the goal of maximizing a consistency score. Our extensive validation on two datasets, MSCOCO and PartiPrompts, shows that OPT2I can boost the initial consistency score by up to 24.9% in terms of DSG score while preserving the FID and increasing the recall between generated and real data. Our work paves the way toward building more reliable and robust T2I systems by harnessing the power of LLMs.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 40 of its 46 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

40 of 46 references.

References · 40

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
28 more
  1. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  2. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  3. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  4. 2023Large Language Models as Optimizerscited by 830
  5. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  6. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  7. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  8. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  9. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  10. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  11. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  12. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  13. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  14. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  15. 2023Larger language models do in-context learning differentlycited by 386
  16. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  17. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  18. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  19. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  20. 2023Large Language Models as General Pattern Machinescited by 196
  21. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  22. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  23. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  24. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
  25. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53
  26. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesiscited by 51
  27. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selectioncited by 40
  28. 2023Transferring Visual Attributes from Natural Language to Verified Image Generationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 39; 2026: 28.

88 papers.

Sort:

By area:

  1. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  2. 2025A Systematic Survey of Automatic Prompt Optimization Techniquescited by 61
  3. 2024Beyond Model Adaptation at Test Time: A Surveycited by 55
  4. 2025SnapMoGen: Human Motion Generation from Expressive Textscited by 44
  5. 2025PromptEnhancer: A Simple Approach to Enhance Text-to-Image Models via Chain-of-Thought Prompt Rewritingcited by 38
  6. 2024Prompt Optimization with Human Feedbackcited by 35
  7. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  8. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  9. 2025T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generationcited by 31
  10. 2025VISTA: A Test-Time Self-Improving Video Generation Agentcited by 26
  11. 2026Diamond Maps: Efficient Reward Alignment via Stochastic Flow Mapscited by 26
  12. 2024Improving Geo-diversity of Generated Images with Contextualized Vendi Score Guidancecited by 24
38 more
  1. 2024Aviary: training language agents on challenging scientific taskscited by 23
  2. 2024Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLMcited by 22
  3. 2025RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learningcited by 22
  4. 2025PromptSculptor: Multi-Agent Based Text-to-Image Prompt Optimizationcited by 19
  5. 2024Efficient and Accurate Prompt Optimization: the Benefit of Memory in Exemplar-Guided Reflectioncited by 18
  6. 2025Maestro: Self-Improving Text-to-Image Generation via Agent Orchestrationcited by 18
  7. 2025Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agentscited by 16
  8. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  9. 2025A Unified Agentic Framework for Evaluating Conditional Image Generationcited by 15
  10. 2025ComfyUI-R1: Exploring Reasoning Models for Workflow Generationcited by 15
  11. 2024Self-Correcting Text-to-Video Generation with Misalignment Detection and Localized Refinementcited by 13
  12. 2024Automated Black-box Prompt Engineering for Personalized Text-to-Image Generationcited by 12
  13. 2024CaPo: Cooperative Plan Optimization for Efficient Embodied Multi-Agent Cooperationcited by 12
  14. 2024Text Embedding is Not All You Need: Attention Control for Text-to-Image Semantic Alignment with Text Self-Attention Mapscited by 12
  15. 2024What makes a good metric? Evaluating automatic metrics for text-to-image consistencycited by 11
  16. 2025ComfyUI-Copilot: An Intelligent Assistant for Automated Workflow Developmentcited by 11
  17. 2026PromptRL: Prompt Matters in RL for Flow-Based Image Generationcited by 11
  18. 2025SAGI: Semantically Aligned and Uncertainty Guided AI Image Inpaintingcited by 10
  19. 2025RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scalingcited by 9
  20. 2025It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Modelscited by 9
  21. 2024Information Theoretic Text-to-Image Alignmentcited by 8
  22. 2024CoCoNO: Attention Contrast-and-Complete for Initial Noise Optimization in Text-to-Image Synthesiscited by 7
  23. 2025DLPO: Towards a Robust, Efficient, and Generalizable Prompt Optimization Framework from a Deep-Learning Perspectivecited by 7
  24. 2026AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompterscited by 7
  25. 2024TIPO: Text to Image with Text Presampling for Prompt Optimizationcited by 6
  26. 2025GenPilot: A Multi-Agent System for Test-Time Prompt Optimization in Image Generationcited by 6
  27. 2026SOAR: Self-Correction for Optimal Alignment and Refinement in Diffusion Modelscited by 6
  28. 2024FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive Prompt Weightingcited by 5
  29. 2025Causally Steered Diffusion for Automated Video Counterfactual Generationcited by 5
  30. 2025Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Modelscited by 5
  31. 2025GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Designcited by 5
  32. 2025Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMscited by 5
  33. 2026A Systematic Post-Train Framework for Video Generationcited by 5
  34. 2024Taming Text-to-Image Synthesis for Novices: User-centric Prompt Generation via Multi-turn Guidancecited by 4
  35. 2024Fast Prompt Alignment for Text-to-Image Generationcited by 4
  36. 2025World-To-Image: Grounding Text-to-Image Generation with Agent-Driven World Knowledgecited by 4
  37. 2024Model-Agnostic Human Preference Inversion in Diffusion Modelscited by 3
  38. 2025WAVE: Warp-Based View Guidance for Consistent Novel View Synthesis Using a Single Imagecited by 3

50 most cited of 88 All 88 (JSON)