mlsanityData to 14 Oct 2026

GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation

arXiv 2406.13743 ↗First posted 19 Jun 2024cs.CV cs.AI cs.CLcited by 135

Not on the map. Why?

Cited by 22 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While text-to-visual models now produce photo-realistic images and videos, they struggle with compositional text prompts involving attributes, relationships, and higher-order reasoning such as logic and comparison. In this work, we conduct an extensive human study on GenAI-Bench to evaluate the performance of leading image and video generation models in various aspects of compositional text-to-visual generation. We also compare automated evaluation metrics against our collected human ratings and find that VQAScore -- a metric measuring the likelihood that a VQA model views an image as accurately depicting the prompt -- significantly outperforms previous metrics such as CLIPScore. In addition, VQAScore can improve generation in a black-box manner (without finetuning) via simply ranking a few (3 to 9) candidate images. Ranking by VQAScore is 2x to 3x more effective than other scoring methods like PickScore, HPSv2, and ImageReward at improving human alignment ratings for DALL-E 3 and Stable Diffusion, especially on compositional prompts that require advanced visio-linguistic reasoning. We release a new GenAI-Rank benchmark with over 40,000 human ratings to evaluate scoring metrics on ranking images generated from the same prompt. Lastly, we discuss promising areas for improvement in VQAScore, such as addressing fine-grained visual details. We will release all human ratings (over 80,000) to facilitate scientific benchmarking of both generative models and automated metrics.

Key facts. Cited by 135 papers in the map (data through 14 Oct 2026). Builds on 66 of its 83 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

66 of 83 references.

References · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  12. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  2. 2016Improved Techniques for Training GANscited by 2,970
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  5. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  6. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  9. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,178
  10. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  11. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  12. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  13. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  14. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  15. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  16. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  17. 2023ModelScope Text-to-Video Technical Reportcited by 658
  18. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 648
  19. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  20. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  21. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  22. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  23. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  24. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  25. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  26. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 288
  27. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 279
  28. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 233
  29. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  30. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 171
  31. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  32. 2024GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generationcited by 153
  33. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  34. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 108
  35. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 90
  36. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 87
  37. 2020On the Value of Out-of-Distribution Testing: An Example of Goodhart's Lawcited by 74
  38. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 74

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 62; 2026: 61.

135 papers.

Sort:

By area:

  1. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
  2. 2025Memory in the Age of AI Agentscited by 303
  3. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 204
  4. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 97
  5. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  6. 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 87
  7. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  8. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 69
  9. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 57
  10. 2025GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluationcited by 52
  11. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 47
  12. 2026Image Generators are Generalist Vision Learnerscited by 43
38 more
  1. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 41
  2. 2025Interleaving Reasoning for Better Text-to-Image Generationcited by 41
  3. 2024EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluationcited by 39
  4. 2025T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generationcited by 38
  5. 2025FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmarkcited by 38
  6. 2025Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modelingcited by 37
  7. 2025Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?cited by 34
  8. 2025R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generationcited by 32
  9. 2026Improved Baselines with Representation Autoencoderscited by 31
  10. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 25
  11. 2025VideoGen-Eval: Agent-based System for Video Generation Evaluationcited by 23
  12. 2025Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferencescited by 22
  13. 2026Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluationcited by 22
  14. 2025SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generationcited by 21
  15. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Imagecited by 20
  16. 2025Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Modelscited by 18
  17. 2025MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Modelscited by 17
  18. 2024Evaluating Image Hallucination in Text-to-Image Generation with Question-Answeringcited by 16
  19. 2025MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoningcited by 16
  20. 2024Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through f-divergence Minimizationcited by 15
  21. 2024T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Conceptscited by 14
  22. 2025Diffusion Models Through a Global Lens: Are They Culturally Inclusive?cited by 14
  23. 2025ToolMem: Enhancing Multimodal Agents with Learnable Tool Capability Memorycited by 14
  24. 2025Dynamic Classifier-Free Diffusion Guidance via Online Feedbackcited by 13
  25. 2024Progressive Compositionality in Text-to-Image Generative Modelscited by 12
  26. 2024Detecting Human Artifacts from Text-to-Image Modelscited by 12
  27. 2025Understanding-in-Generation: Reinforcing Generative Capability of Unified Model via Infusing Understanding into Generationcited by 11
  28. 2026TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMscited by 11
  29. 2026BizGenEval: A Systematic Benchmark for Commercial Visual Content Generationcited by 11
  30. 2025A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generationcited by 10
  31. 2025NTIRE 2025 challenge on Text to Image Generation Model Quality Assessmentcited by 8
  32. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  33. 2025UniFusion: Vision-Language Model as Unified Encoder in Image Generationcited by 8
  34. 2026AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompterscited by 8
  35. 2026Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generationcited by 8
  36. 2025GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoningcited by 7
  37. 2025Visual Persona: Foundation Model for Full-Body Human Customizationcited by 7
  38. 2025FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judgescited by 7

50 most cited of 135 All 135 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating the Evaluators: Metrics for Compositional Text-to-Image Generation
  2. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation
  3. 2026Iterative Refinement Improves Compositional Image Generation
  4. 2024ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficulty
  5. 2025UniGenBench++: A Unified Semantic Evaluation Benchmark for Text-to-Image Generation
  6. 2025Draw ALL Your Imagine: A Holistic Benchmark and Agent Framework for Complex Instruction-based Image Generation
  7. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation
  8. 2025CompAlign: Improving Compositional Text-to-Image Generation with a Complex Benchmark and Fine-Grained Feedback