mlsanityData to 7 Oct 2026

Evaluating Text-to-Visual Generation with Image-to-Text Generation

arXiv 2404.01291 ↗First posted 1 Apr 2024cs.CV cs.AI cs.CLcited by 465

Not on the map. Why?

Cited by 100 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Despite significant progress in generative AI, comprehensive evaluation remains challenging because of the lack of effective metrics and standardized benchmarks. For instance, the widely-used CLIPScore measures the alignment between a (generated) image and text prompt, but it fails to produce reliable scores for complex prompts involving compositions of objects, attributes, and relations. One reason is that text encoders of CLIP can notoriously act as a "bag of words", conflating prompts such as "the horse is eating the grass" with "the grass is eating the horse". To address this, we introduce the VQAScore, which uses a visual-question-answering (VQA) model to produce an alignment score by computing the probability of a "Yes" answer to a simple "Does this figure show '{text}'?" question. Though simpler than prior art, VQAScore computed with off-the-shelf models produces state-of-the-art results across many (8) image-text alignment benchmarks. We also compute VQAScore with an in-house model that follows best practices in the literature. For example, we use a bidirectional image-question encoder that allows image embeddings to depend on the question being asked (and vice versa). Our in-house model, CLIP-FlanT5, outperforms even the strongest baselines that make use of the proprietary GPT-4V. Interestingly, although we train with only images, VQAScore can also align text with video and 3D models. VQAScore allows researchers to benchmark text-to-visual generation using complex texts that capture the compositional structure of real-world prompts. We introduce GenAI-Bench, a more challenging benchmark with 1,600 compositional text prompts that require parsing scenes, objects, attributes, relationships, and high-order reasoning like comparison and logic. GenAI-Bench also offers over 15,000 human ratings for leading image and video generation models such as Stable Diffusion, DALL-E 3, and Gen2.

Key facts. Cited by 465 papers in the map (data through 7 Oct 2026). Builds on 79 of its 94 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

79 of 94 references.

References · 79

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2016Improved Techniques for Training GANscited by 2,938
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  12. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  13. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  14. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  15. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  16. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  17. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  18. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  19. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  20. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  21. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  22. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  23. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  24. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  25. 2023ModelScope Text-to-Video Technical Reportcited by 651
  26. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  27. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  28. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  29. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  30. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  31. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  32. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  33. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  34. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  35. 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 438
  36. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  37. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  38. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 73; 2025: 232; 2026: 160.

465 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  3. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  4. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  5. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  6. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  7. 2024An Introduction to Vision-Language Modelingcited by 175
  8. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 167
  9. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  10. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
  11. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  12. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
38 more
  1. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  2. 2024ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimizationcited by 115
  3. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  4. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  5. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  6. 2025Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generationcited by 65
  7. 2024ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhancecited by 64
  8. 2025TIIF-Bench: How Does Your T2I Model Follow Your Instructions?cited by 64
  9. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 62
  10. 2025Towards Understanding Camera Motions in Any Videocited by 62
  11. 2024ConceptMix: A Compositional Image Generation Benchmark with Controllable Difficultycited by 60
  12. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 59
  13. 2025EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editingcited by 59
  14. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 55
  15. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 53
  16. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  17. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
  18. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  19. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  20. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  21. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabulariescited by 42
  22. 2025GenEval 2: Addressing Benchmark Drift in Text-to-Image Evaluationcited by 42
  23. 2025Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modelingcited by 41
  24. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 40
  25. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them Firstcited by 39
  26. 2024A-Bench: Are LMMs Masters at Evaluating AI-generated Images?cited by 39
  27. 2025T2I-ReasonBench: Benchmarking Reasoning-Informed Text-to-Image Generationcited by 38
  28. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 37
  29. 2025Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcingcited by 37
  30. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 35
  31. 2025Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimizationcited by 35
  32. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  33. 2025AlignTok: Aligning Visual Foundation Encoders to Tokenizers for Diffusion Modelscited by 35
  34. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  35. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  36. 2026Inference-time Physics Alignment of Video Generative Models with Latent World Modelscited by 32
  37. 2025R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generationcited by 31
  38. 2025T2I-Copilot: A Training-Free Multi-Agent Text-to-Image System for Enhanced Prompt Interpretation and Interactive Generationcited by 31

50 most cited of 465 All 465 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Flexible Evaluation for Generative Visual Question Answering
  2. 2023VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning
  3. 2023Generative Visual Question Answering
  4. 2025Text-VQA Aug: Pipelined Harnessing of Large Multimodal Models for Automated Synthesis
  5. 2024A Comprehensive Survey on Visual Question Answering Datasets and Algorithms
  6. 2026DEPICT: Scoring Text-to-Image Alignment by Answer Agreement
  7. 2024Enhancing Visual Question Answering through Question-Driven Image Captions as Prompts
  8. 2025Test-time Prompt Refinement for Text-to-Image Models