mlsanityData to 14 Oct 2026

Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation

arXiv 2304.01816 ↗First posted 4 Apr 2023cs.CVcited by 68

Not on the map. Why?

Abstract (arXiv)

Human evaluation is critical for validating the performance of text-to-image generative models, as this highly cognitive process requires deep comprehension of text and images. However, our survey of 37 recent papers reveals that many works rely solely on automatic measures (e.g., FID) or perform poorly described human evaluations that are not reliable or repeatable. This paper proposes a standardized and well-defined human evaluation protocol to facilitate verifiable and reproducible human evaluation in future works. In our pilot data collection, we experimentally show that the current automatic measures are incompatible with human perception in evaluating the performance of the text-to-image generation results. Furthermore, we provide insights for designing human evaluation experiments reliably and conclusively. Finally, we make several resources publicly available to the community to facilitate easy and fast implementations.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Builds on 44 of its 68 references that are in the map. First posted on arXiv 4 Apr 2023.

Builds on

44 of 68 references.

References · 44

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  7. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  8. 2016Improved Techniques for Training GANscited by 2,970
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
32 more
  1. 2018Demystifying MMD GANscited by 1,293
  2. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  3. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  4. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  5. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  6. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  7. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  8. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  9. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  10. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  11. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  12. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  13. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  14. 2021How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Modelscited by 196
  15. 2019Controllable Text-to-Image Generationcited by 187
  16. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  17. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 162
  18. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 128
  19. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  20. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  21. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 97
  22. 2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
  23. 2018Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Networkcited by 64
  24. 2019Semantics Disentangling for Text-to-Image Generationcited by 61
  25. 2019HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Modelscited by 59
  26. 2021Text to Image Generation with Semantic-Spatial Aware GANcited by 53
  27. 2021DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesiscited by 45
  28. 2022StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesiscited by 33
  29. 2018An Improved Evaluation Framework for Generative Adversarial Networkscited by 26
  30. 2021On Training Sample Memorization: Lessons from Benchmarking Generative Modeling with a Large-scale Competitioncited by 18
  31. 2019Dual Adversarial Inference for Text-to-Image Synthesiscited by 17
  32. 2021TISE: Bag of Metrics for Text-to-Image Synthesis Evaluationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 23; 2025: 16; 2026: 12.

68 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  2. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  3. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  4. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 205
  5. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  6. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 158
  7. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  8. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 141
  9. 2023Rich Human Feedback for Text-to-Image Generationcited by 141
  10. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 135
  11. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  12. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 119
38 more
  1. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 82
  2. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 75
  3. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 57
  4. 2024Towards A Better Metric for Text-to-Video Generationcited by 51
  5. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selectioncited by 40
  6. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 36
  7. 2025Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?cited by 34
  8. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  9. 2023Fair Text-to-Image Diffusion via Fair Mappingcited by 31
  10. 2024GAIA: Rethinking Action Quality Assessment for AI-Generated Videoscited by 30
  11. 2024Erasing Concepts from Text-to-Image Diffusion Models with Few-shot Unlearningcited by 29
  12. 2023X-IQE: eXplainable Image Quality Evaluation for Text-to-Image Generation with Visual Large Language Modelscited by 28
  13. 2023Stable Diffusion Exposed: Gender Bias from Prompt to Imagecited by 25
  14. 2023Exploring the Naturalness of AI-Generated Imagescited by 22
  15. 2025NarrLV: Towards a Comprehensive Narrative-Centric Evaluation for Long Video Generationcited by 22
  16. 2024EVALALIGN: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Modelscited by 20
  17. 2026Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Modelscited by 18
  18. 2023PKU-I2IQA: An Image-to-Image Quality Assessment Database for AI Generated Imagescited by 14
  19. 2024PFStorer: Personalized Face Restoration and Super-Resolutioncited by 13
  20. 2024Detecting Human Artifacts from Text-to-Image Modelscited by 12
  21. 2025Direct Motion Models for Assessing Generated Videoscited by 10
  22. 2025LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretationcited by 10
  23. 2025One-Step Offline Distillation of Diffusion-based Models via Koopman Modelingcited by 9
  24. 2024Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesiscited by 7
  25. 2024Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Surveycited by 6
  26. 2023Let's ViCE! Mimicking Human Cognitive Behavior in Image Generation Evaluationcited by 5
  27. 2023Hierarchical Evaluation Framework: Best Practices for Human Evaluationcited by 5
  28. 2023Filter & Align: Leveraging Human Knowledge to Curate Image-Text Datacited by 4
  29. 2024Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability,Reproducibility, and Practicalitycited by 4
  30. 2025HiCoGen: Hierarchical Compositional Text-to-Image Generation in Diffusion Models via Reinforcement Learningcited by 4
  31. 2026Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributionscited by 4
  32. 2023The Challenges of Image Generation Models in Generating Multi-Component Imagescited by 3
  33. 2024Greedy Growing Enables High-Resolution Pixel-Based Diffusion Modelscited by 3
  34. 2024A Framework for Critical Evaluation of Text-to-Image Models: Integrating Art Historical Analysis, Artistic Exploration, and Critical Prompt Engineeringcited by 3
  35. 2024Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Modelscited by 2
  36. 2024SLayR: Scene Layout Generation with Rectified Flowcited by 2
  37. 2025A Speech-to-Video Synthesis Approach Using Spatio-Temporal Diffusion for Vocal Tract MRIcited by 2
  38. 2025Multi-interaction TTS toward professional recording reproductioncited by 2

50 most cited of 68 All 68 (JSON)