mlsanityData to 7 Oct 2026

LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation

arXiv 2305.11116 ↗First posted 18 May 2023cs.CV cs.CLcited by 85

Not on the map. Why?

Abstract (arXiv)

Existing automatic evaluation on text-to-image synthesis can only provide an image-text matching score, without considering the object-level compositionality, which results in poor correlation with human judgments. In this work, we propose LLMScore, a new framework that offers evaluation scores with multi-granularity compositionality. LLMScore leverages the large language models (LLMs) to evaluate text-to-image models. Initially, it transforms the image into image-level and object-level visual descriptions. Then an evaluation instruction is fed into the LLMs to measure the alignment between the synthesized image and the text, ultimately generating a score accompanied by a rationale. Our substantial analysis reveals the highest correlation of LLMScore with human judgments on a wide range of datasets (Attribute Binding Contrast, Concept Conjunction, MSCOCO, DrawBench, PaintSkills). Notably, our LLMScore achieves Kendall's tau correlation with human evaluations that is 58.8% and 31.2% higher than the commonly-used text-image matching metrics CLIP and BLIP, respectively.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 49 of its 53 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

49 of 53 references.

References · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2017Proximal Policy Optimization Algorithmscited by 14,441
  9. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
37 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2016Improved Techniques for Training GANscited by 2,938
  7. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  13. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  14. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  15. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  16. 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
  17. 2022PAL: Program-aided Language Modelscited by 635
  18. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  19. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  20. 2022Blended Latent Diffusioncited by 430
  21. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  22. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  23. 2018Assessing Generative Models via Precision and Recallcited by 401
  24. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  25. 2023GPTScore: Evaluate as You Desirecited by 357
  26. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  27. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  28. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  29. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  30. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  31. 2023Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillingscited by 68
  32. 2023Multimodal Procedural Planning via Dual Text-Image Promptingcited by 40
  33. 2023Tuning computer vision models with task rewardscited by 36
  34. 2022Mutual Information Divergence: A Unified Metric for Multimodal Generative Modelscited by 35
  35. 2022StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesiscited by 33
  36. 2022Visualize Before You Write: Imagination-Guided Open-Ended Text Generationcited by 30
  37. 2022Imagination-Augmented Natural Language Understandingcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 46; 2025: 19; 2026: 9.

85 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  3. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  4. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  5. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  6. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  7. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  8. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  9. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  10. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  11. 2023Divide & Bind Your Attention for Improved Generative Semantic Nursingcited by 80
  12. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
38 more
  1. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 53
  2. 2024Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?cited by 52
  3. 2023Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Promptscited by 51
  4. 2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
  5. 2024TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generationcited by 42
  6. 2024Reward Guided Latent Consistency Distillationcited by 35
  7. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  8. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  9. 2025R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generationcited by 31
  10. 2024Beyond Aesthetics: Cultural Competence in Text-to-Image Modelscited by 30
  11. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  12. 2024Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertaintycited by 23
  13. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?cited by 20
  14. 2024EVALALIGN: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Modelscited by 20
  15. 2024RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Modelscited by 18
  16. 2024A Survey on Quality Metrics for Text-to-Image Generationcited by 17
  17. 2024VersaT2I: Improving Text-to-Image Models with Versatile Rewardcited by 17
  18. 2024Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)cited by 17
  19. 2024Energy-based Automated Model Evaluationcited by 16
  20. 2024Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Rewardcited by 16
  21. 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimizationcited by 15
  22. 2025REALEDIT: Reddit Edits As a Large-scale Empirical Dataset for Image Transformationscited by 15
  23. 2025A Unified Agentic Framework for Evaluating Conditional Image Generationcited by 15
  24. 2025Science-T2I: Addressing Scientific Illusions in Image Synthesiscited by 15
  25. 2025CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metricscited by 14
  26. 2024Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmarkcited by 13
  27. 2023A Contrastive Compositional Benchmark for Text-to-Image Synthesis: A Study with Unified Text-to-Image Fidelity Metricscited by 12
  28. 2024T2I-FactualBench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Conceptscited by 12
  29. 2024GT23D-Bench: A Comprehensive General Text-to-3D Generation Benchmarkcited by 11
  30. 2024Adaptive Mixed-Scale Feature Fusion Network for Blind AI-Generated Image Quality Assessmentcited by 10
  31. 2024Evaluating Hallucination in Text-to-Image Diffusion Models with Scene-Graph based Question-Answering Agentcited by 9
  32. 2025Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generationcited by 9
  33. 2025Balancing Preservation and Modification: A Region and Semantic Aware Metric for Instruction-Based Image Editingcited by 9
  34. 2024Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generationcited by 8
  35. 2024EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Modelscited by 8
  36. 2024ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Modelscited by 8
  37. 2024GraphAgent: Agentic Graph Language Assistantcited by 8
  38. 2023ChatTraffic: Text-to-Traffic Generation via Diffusion Modelcited by 7

50 most cited of 85 All 85 (JSON)