mlsanityData to 7 Oct 2026

If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selection

arXiv 2305.13308 ↗First posted 22 May 2023cs.CVcited by 40

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Despite their impressive capabilities, diffusion-based text-to-image (T2I) models can lack faithfulness to the text prompt, where generated images may not contain all the mentioned objects, attributes or relations. To alleviate these issues, recent works proposed post-hoc methods to improve model faithfulness without costly retraining, by modifying how the model utilizes the input prompt. In this work, we take a step back and show that large T2I diffusion models are more faithful than usually assumed, and can generate images faithful to even complex prompts without the need to manipulate the generative process. Based on that, we show how faithfulness can be simply treated as a candidate selection problem instead, and introduce a straightforward pipeline that generates candidate images for a text prompt and picks the best one according to an automatic scoring system that can leverage already existing T2I evaluation metrics. Quantitative comparisons alongside user studies on diverse benchmarks show consistently improved faithfulness over post-hoc enhancement methods, with comparable or lower computational cost. Code is available at urlhttps://github.com/ExplainableML/ImageSelect.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 62 of its 71 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

62 of 71 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models11 Apr
  2. 2023Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis7 Apr
  3. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models31 Jan
  4. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  5. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  13. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  14. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  15. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  16. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  17. 2016Generative Adversarial Text to Image Synthesiscited by 735
  18. 2022Autoregressive Image Generation using Residual Quantizationcited by 706
  19. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  20. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  21. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  22. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  23. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  24. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  25. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  26. 2022Visual Spatial Reasoningcited by 288
  27. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  28. 2023StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesiscited by 212
  29. 2023Teaching CLIP to Count to Tencited by 194
  30. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  31. 2015Generating Images from Captions with Attentioncited by 135
  32. 2023Social Biases through the Text-to-Image Generation Lenscited by 126
  33. 2016Learning What and Where to Drawcited by 105
  34. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  35. 2022How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions?cited by 92
  36. 2022Character-Aware Models Improve Visual Text Renderingcited by 90
  37. 2021Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer visioncited by 76
  38. 2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 66

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 13; 2025: 14; 2026: 9.

40 papers.

Sort:

By area:

  1. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 225
  2. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  3. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 102
  4. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
  5. 2024Not All Noises Are Created Equally:Diffusion Noise Selection and Optimizationcited by 70
  6. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
  7. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 35
  8. 2024Token Merging for Training-Free Semantic Binding in Text-to-Image Synthesiscited by 35
  9. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  10. 2026Diamond Maps: Efficient Reward Alignment via Stochastic Flow Mapscited by 26
  11. 2025PromptSculptor: Multi-Agent Based Text-to-Image Prompt Optimizationcited by 19
  12. 2024VersaT2I: Improving Text-to-Image Models with Versatile Rewardcited by 17
28 more
  1. 2024PerCo (SD): Open Perceptual Compressioncited by 16
  2. 2023Correcting Diffusion Generation through Resamplingcited by 15
  3. 2024Progressive Compositionality in Text-to-Image Generative Modelscited by 11
  4. 2025Dynamic Classifier-Free Diffusion Guidance via Online Feedbackcited by 11
  5. 2025It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Modelscited by 9
  6. 2023Diverse Diffusion: Enhancing Image Diversity in Text-to-Image Generationcited by 8
  7. 2024Information Theoretic Text-to-Image Alignmentcited by 8
  8. 2024Removing Distributional Discrepancies in Captions Improves Image-Text Alignmentcited by 8
  9. 2024Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!cited by 8
  10. 2025RewardSDS: Aligning Score Distillation via Reward-Weighted Samplingcited by 4
  11. 2025RAGME: Retrieval Augmented Video Generation for Enhanced Motion Realismcited by 4
  12. 2025Evaluating the Evaluators: Metrics for Compositional Text-to-Image Generationcited by 4
  13. 2025AutoRefiner: Improving Autoregressive Video Diffusion Models via Reflective Refinement Over the Stochastic Sampling Pathcited by 4
  14. 2024Optimizing Resource Consumption in Diffusion Models through Hallucination Early Detectioncited by 3
  15. 2026Aligning Few-Step Generative Models by Amortizing Sample-based Variational Inferencecited by 3
  16. 2024CUPID: Contextual Understanding of Prompt-conditioned Image Distributionscited by 2
  17. 2025Why Settle for Mid: A Probabilistic Viewpoint to Spatial Relationship Alignment in Text-to-image Modelscited by 2
  18. 2025CARINOX: Inference-time Scaling with Category-Aware Reward-based Initial Noise Optimization and Explorationcited by 2
  19. 2026Bias at the End of the Scorecited by 1
  20. 2025Dual Caption Preference Optimization for Diffusion Modelscited by 0
  21. 2025Improving Compositional Generation with Diffusion Models Using Lift Scorescited by 0
  22. 2025Blending Concepts with Text-to-Image Diffusion Modelscited by 0
  23. 2026Personalizing Text-to-Image Generation to Individual Tastecited by 0
  24. 2026Hallucination Early Detection in Diffusion Modelscited by 0
  25. 2026Follow the Mean: Reference-Guided Flow Matchingcited by 0
  26. 2026Reasoning with Image Generationcited by 0
  27. 2026Weeding Out Bad Seeds: Initial-Noise-Robust Unlearning for Text-to-Image Diffusion Modelscited by 0
  28. 2026Posterior sampling by source-space MCMC via prior-based few-step transport mapscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Text-to-image Diffusion Models in Generative AI: A Survey
  2. 2025Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
  3. 2026Na"ive PAINE: Lightweight Text-to-Image Generation Improvement with Prompt Evaluation
  4. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Models
  5. 2024Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering
  6. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
  7. 2025Text-to-Image Diffusion Models Cannot Count, and Prompt Refinement Cannot Help
  8. 2024All Seeds Are Not Equal: Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds