mlsanityData to 14 Oct 2026

Holistic Evaluation of Text-To-Image Models

arXiv 2311.04287 ↗First posted 7 Nov 2023cs.CV cs.LGcited by 160

Not on the map. Why?

Cited by 26 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The stunning qualitative improvement of recent text-to-image models has led to their widespread attention and adoption. However, we lack a comprehensive quantitative understanding of their capabilities and risks. To fill this gap, we introduce a new benchmark, Holistic Evaluation of Text-to-Image Models (HEIM). Whereas previous evaluations focus mostly on text-image alignment and image quality, we identify 12 aspects, including text-image alignment, image quality, aesthetics, originality, reasoning, knowledge, bias, toxicity, fairness, robustness, multilinguality, and efficiency. We curate 62 scenarios encompassing these aspects and evaluate 26 state-of-the-art text-to-image models on this benchmark. Our results reveal that no single model excels in all aspects, with different models demonstrating different strengths. We release the generated images and human evaluation results for full transparency at https://crfm.stanford.edu/heim/v1.1.0 and the code at https://github.com/stanford-crfm/helm, which is integrated with the HELM codebase.

Key facts. Cited by 160 papers in the map (data through 14 Oct 2026). Builds on 49 of its 100 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

49 of 100 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
37 more
  1. 2014Going Deeper with Convolutionscited by 4,241
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2016Improved Techniques for Training GANscited by 2,970
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,375
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  9. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  10. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  11. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  12. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  13. 2023Extracting Training Data from Diffusion Modelscited by 732
  14. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  15. 2023Erasing Concepts from Diffusion Modelscited by 539
  16. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  17. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  18. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  19. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  20. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  21. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  22. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 317
  23. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 307
  24. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 265
  25. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  26. 2022Optimizing Prompts for Text-to-Image Generationcited by 207
  27. 2016Learning Deep Representations of Fine-grained Visual Descriptionscited by 182
  28. 2019Classification Accuracy Score for Conditional Generative Modelscited by 136
  29. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  30. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 112
  31. 2022Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domainscited by 86
  32. 2023Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generationcited by 68
  33. 2021Understanding and Creating Art with AI: Review and Outlookcited by 60
  34. 2019HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Modelscited by 59
  35. 2022Testing Relational Understanding in Text-Guided Image Generationcited by 59
  36. 2022DALL-E 2 Fails to Reliably Capture Common Syntactic Processescited by 31
  37. 2023Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 83; 2025: 35; 2026: 31.

160 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  2. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  3. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  4. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 201
  5. 2023MMA-Diffusion: MultiModal Attack on Diffusion Modelscited by 194
  6. 2023Rich Human Feedback for Text-to-Image Generationcited by 141
  7. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 135
  8. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  9. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  10. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 108
  11. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  12. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 82
38 more
  1. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  2. 2024Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigationcited by 66
  3. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 66
  4. 2024T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Modelscited by 66
  5. 2024GuardT2I: Defending Text-to-Image Models from Adversarial Promptscited by 62
  6. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 62
  7. 2023Shadows Don't Lie and Lines Can't Bend! Generative Models don't know Projective Geometry...for nowcited by 61
  8. 2024ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficultycited by 61
  9. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  10. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 48
  11. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 45
  12. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  13. 2025AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Cloudscited by 40
  14. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 36
  15. 2024PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Modelscited by 36
  16. 2025FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Modelcited by 32
  17. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 28
  18. 2023Diffusion Illusions: Hiding Images in Plain Sightcited by 26
  19. 2023A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model's Accuracy for Question Answering on Enterprise SQL Databasescited by 25
  20. 2023Stable Diffusion Exposed: Gender Bias from Prompt to Imagecited by 25
  21. 2024Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmarkcited by 23
  22. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 22
  23. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  24. 2024EVALALIGN: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Modelscited by 20
  25. 2024Fairness and Bias in Multimodal AI: A Surveycited by 20
  26. 2024A Survey on Quality Metrics for Text-to-Image Generationcited by 18
  27. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  28. 2024VersaT2I: Improving Text-to-Image Models with Versatile Rewardcited by 17
  29. 2025Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusioncited by 17
  30. 2025MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Modelscited by 17
  31. 2024Towards Geographic Inclusion in the Evaluation of Text-to-Image Modelscited by 16
  32. 2024K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferencescited by 16
  33. 2026Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Modelscited by 16
  34. 2024Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help Youcited by 15
  35. 2024Boosting Alignment for Post-Unlearning Text-to-Image Generative Modelscited by 15
  36. 2024Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddingscited by 15
  37. 2024T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Conceptscited by 14
  38. 2025The Measurement Imbalance in Agentic AI Evaluation Undermines Industry Productivity Claimscited by 14

50 most cited of 160 All 160 (JSON)