mlsanityData to 7 Oct 2026

HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models

arXiv 2304.05390 ↗First posted 11 Apr 2023cs.CV cs.AI cs.LGcited by 94

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

In recent years, Text-to-Image (T2I) models have been extensively studied, especially with the emergence of diffusion models that achieve state-of-the-art results on T2I synthesis tasks. However, existing benchmarks heavily rely on subjective human evaluation, limiting their ability to holistically assess the model's capabilities. Furthermore, there is a significant gap between efforts in developing new T2I architectures and those in evaluation. To address this, we introduce HRS-Bench, a concrete evaluation benchmark for T2I models that is Holistic, Reliable, and Scalable. Unlike existing bench-marks that focus on limited aspects, HRS-Bench measures 13 skills that can be categorized into five major categories: accuracy, robustness, generalization, fairness, and bias. In addition, HRS-Bench covers 50 scenarios, including fashion, animals, transportation, food, and clothes. We evaluate nine recent large-scale T2I models using metrics that cover a wide range of skills. A human evaluation aligned with 95% of our evaluations on average was conducted to probe the effectiveness of HRS-Bench. Our experiments demonstrate that existing models often struggle to generate images with the desired count of objects, visual text, or grounded emotions. We hope that our benchmark help ease future text-to-image generation research. The code and data are available at https://eslambakr.github.io/hrsbench.github.io

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 52 of its 77 references that are in the map. First posted on arXiv 11 Apr 2023.

Builds on

52 of 77 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  2. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models26 Oct
  3. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
34 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  6. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  7. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  8. 2016Generative Adversarial Text to Image Synthesiscited by 735
  9. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  10. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  11. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  12. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  13. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  14. 2017On Fairness and Calibrationcited by 406
  15. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  16. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  17. 2018A Note on the Inception Scorecited by 225
  18. 2018Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognitioncited by 130
  19. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  20. 2021Simple multi-dataset detectioncited by 126
  21. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  22. 2019ICDAR 2019 Competition on Large-scale Street View Text with Partial Labeling -- RRC-LSVTcited by 121
  23. 2018TextSnake: A Flexible Representation for Detecting Text of Arbitrary Shapescited by 114
  24. 2021ArtEmis: Affective Language for Visual Artcited by 103
  25. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  26. 2016Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmarkcited by 71
  27. 2022Human Evaluation of Text-to-Image Models on a Multi-Task Benchmarkcited by 41
  28. 2022It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image Captioning by Contrastive Data Collectioncited by 36
  29. 2022ComCLIP: Training-Free Compositional Image and Text Matchingcited by 26
  30. 2023Auditing Gender Presentation Differences in Text-to-Image Modelscited by 22
  31. 2022Affection: Learning Affective Explanations for Real-World Visual Datacited by 19
  32. 2022ArtELingo: A Million Emotion Annotations of WikiArt with Emphasis on Diversity over Language and Culturecited by 17
  33. 2021TISE: Bag of Metrics for Text-to-Image Synthesis Evaluationcited by 13
  34. 2022On the Robustness of Quality Measures for GANscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 40; 2025: 26; 2026: 10.

94 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  3. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  4. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  5. 2023Grounded Text-to-Image Synthesis with Attention Refocusingcited by 163
  6. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 151
  7. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  8. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  9. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
  10. 2023Expressive Text-to-Image Generation with Rich Textcited by 89
  11. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  12. 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 81
38 more
  1. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  2. 2024Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigationcited by 66
  3. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  4. 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 60
  5. 2024ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficultycited by 60
  6. 2023T3Bench: Benchmarking Current Progress in Text-to-3D Generationcited by 56
  7. 2024Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?cited by 52
  8. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
  9. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selectioncited by 40
  10. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  11. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 35
  12. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  13. 2023R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generationcited by 29
  14. 2023Diffusion Model for Camouflaged Object Detectioncited by 27
  15. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  16. 2023Stable Diffusion Exposed: Gender Bias from Prompt to Imagecited by 25
  17. 2023ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generationscited by 24
  18. 2024GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantationcited by 24
  19. 2023LoCo: Locally Constrained Training-Free Layout-to-Image Synthesiscited by 21
  20. 2024FAIntbench: A Holistic and Precise Benchmark for Bias Evaluation in Text-to-Image Modelscited by 21
  21. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  22. 2024EVALALIGN: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Modelscited by 20
  23. 2023Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generationcited by 19
  24. 2026Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluationcited by 16
  25. 2025Psi-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Modelscited by 15
  26. 2025Unified Multimodal Models as Auto-Encoderscited by 13
  27. 2023A Contrastive Compositional Benchmark for Text-to-Image Synthesis: A Study with Unified Text-to-Image Fidelity Metricscited by 12
  28. 2024Discriminative Probing and Tuning for Text-to-Image Generationcited by 11
  29. 2024FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Modelscited by 11
  30. 2024Efficient Lifelong Model Evaluation in an Era of Rapid Progresscited by 10
  31. 2025LatexBlend: Scaling Multi-concept Customized Generation with Latent Textual Blendingcited by 10
  32. 2025ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generationcited by 10
  33. 2025Exploring Bias in over 100 Text-to-Image Generative Modelscited by 9
  34. 2024Attention Overlap Is Responsible for The Entity Missing Problem in Text-to-image Diffusion Models!cited by 8
  35. 2024Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluationcited by 7
  36. 2024IDEA-Bench: How Far are Generative Models from Professional Designing?cited by 7
  37. 2024CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Modelscited by 7
  38. 2025Temporal Concept Dynamics in Diffusion Models via Prompt-Conditioned Interventionscited by 7

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?
  2. 2024BIGbench: A Unified Benchmark for Evaluating Multi-dimensional Social Biases in Text-to-Image Models
  3. 2024EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation
  4. 2025Multimodal Language Models as Text-to-Image Model Evaluators
  5. 2025VISTAR:A User-Centric and Role-Driven Benchmark for Text-to-Image Evaluation
  6. 2026i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models
  7. 2026T2I-BiasBench: A Multi-Metric Framework for Auditing Demographic and Cultural Bias in Text-to-Image Models
  8. 2025TIT-Score: Evaluating Long-Prompt Based Text-to-Image Alignment via Text-to-Image-to-Text Consistency