mlsanityData to 7 Oct 2026

GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment

arXiv 2310.11513 ↗First posted 17 Oct 2023cs.CV cs.LGcited by 946

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Recent breakthroughs in diffusion models, multimodal pretraining, and efficient finetuning have led to an explosion of text-to-image generative models. Given human evaluation is expensive and difficult to scale, automated methods are critical for evaluating the increasingly large number of new models. However, most current automated evaluation metrics like FID or CLIPScore only offer a holistic measure of image quality or image-text alignment, and are unsuited for fine-grained or instance-level analysis. In this paper, we introduce GenEval, an object-focused framework to evaluate compositional image properties such as object co-occurrence, position, count, and color. We show that current object detection models can be leveraged to evaluate text-to-image models on a variety of generation tasks with strong human agreement, and that other discriminative vision models can be linked to this pipeline to further verify properties like object color. We then evaluate several open-source text-to-image models and analyze their relative generative capabilities on our benchmark. We find that recent models demonstrate significant improvement on these tasks, though they are still lacking in complex capabilities such as spatial relations and attribute binding. Finally, we demonstrate how GenEval might be used to help discover existing failure modes, in order to inform development of the next generation of text-to-image models. Our code to run the GenEval framework is publicly available at https://github.com/djghosh13/geneval.

Key facts. Cited by 946 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 27 of its 40 references that are in the map. First posted on arXiv 17 Oct 2023.

Builds on

27 of 40 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023X-IQE: eXplainable Image Quality Evaluation for Text-to-Image Generation with Visual Large Language Models18 May
  2. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2016Improved Techniques for Training GANscited by 2,938
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  12. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
11 more
  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  2. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  3. 2016Generative Adversarial Text to Image Synthesiscited by 735
  4. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  5. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  6. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  7. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  8. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  9. 2015Generating Images from Captions with Attentioncited by 135
  10. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  11. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 73; 2025: 387; 2026: 484.

946 papers.

Sort:

By area:

  1. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  2. 2025Qwen-Image Technical Reportcited by 973
  3. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  5. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  6. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  7. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  8. 2024OmniGen: Unified Image Generationcited by 388
  9. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  10. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  11. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  12. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
38 more
  1. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  2. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  3. 2025Transfer between Modalities with MetaQueriescited by 231
  4. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 204
  5. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  6. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  7. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  8. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  9. 2024Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Modelscited by 151
  10. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  11. 2024Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokenscited by 147
  12. 2025HunyuanImage 3.0 Technical Reportcited by 138
  13. 2025HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformercited by 131
  14. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  15. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  16. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  17. 2024Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Modelscited by 113
  18. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  19. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  20. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  21. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97
  22. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  23. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 94
  24. 2025Masked Autoencoders Are Effective Tokenizers for Diffusion Modelscited by 93
  25. 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 93
  26. 2025LongCat-Image Technical Reportcited by 92
  27. 2024Golden Noise for Diffusion Models: A Learning Frameworkcited by 91
  28. 2025PixelFlow: Pixel-Space Generative Models with Flowcited by 91
  29. 2025Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learningcited by 88
  30. 2025Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Expertscited by 85
  31. 2025SANA-Sprint: One-Step Diffusion with Continuous-Time Consistency Distillationcited by 85
  32. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  33. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85
  34. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  35. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  36. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 81
  37. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  38. 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 81

50 most cited of 946 All 946 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark
  2. 2023What You See is What You Read? Improving Text-Image Alignment Evaluation
  3. 2023Improving Compositional Text-to-image Generation with Large Vision-Language Models
  4. 2024EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation
  5. 2023Filter & Align: Leveraging Human Knowledge to Curate Image-Text Data
  6. 2025Color Bind: Exploring Color Perception in Text-to-Image Models
  7. 2023SelfEval: Leveraging the discriminative nature of generative models for evaluation
  8. 2023Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback