mlsanityData to 7 Oct 2026

What makes a good metric? Evaluating automatic metrics for text-to-image consistency

arXiv 2412.13989 ↗First posted 18 Dec 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Language models are increasingly being incorporated as components in larger AI systems for various purposes, from prompt optimization to automatic evaluation. In this work, we analyze the construct validity of four recent, commonly used methods for measuring text-to-image consistency - CLIPScore, TIFA, VPEval, and DSG - which rely on language models and/or VQA models as components. We define construct validity for text-image consistency metrics as a set of desiderata that text-image consistency metrics should have, and find that no tested metric satisfies all of them. We find that metrics lack sufficient sensitivity to language and visual properties. Next, we find that TIFA, VPEval and DSG contribute novel information above and beyond CLIPScore, but also that they correlate highly with each other. We also ablate different aspects of the text-image consistency metrics and find that not all model components are strictly necessary, also a symptom of insufficient sensitivity to visual information. Finally, we show that all three VQA-based metrics likely rely on familiar text shortcuts (such as yes-bias in QA) that call their aptitude as quantitative evaluations of model performance into question.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 50 of its 68 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

50 of 68 references.

References · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  9. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2019How Can We Know What Language Models Know?cited by 953
38 more
  1. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  2. 2021ImageNet-21K Pretraining for the Massescited by 607
  3. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  4. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  5. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  6. 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
  7. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  8. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  9. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  10. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  11. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  12. 2019Measurement and Fairnesscited by 299
  13. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  14. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  15. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  16. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 133
  17. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  18. 2019Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editingcited by 108
  19. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  20. 2024Improving Text-to-Image Consistency via Automatic Prompt Optimizationcited by 88
  21. 2021Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Modelscited by 76
  22. 2022CREPE: Open-Domain Question Answering with False Presuppositionscited by 70
  23. 2021Human-Adversarial Visual Question Answeringcited by 57
  24. 2023Visual Programming for Text-to-Image Generation and Evaluationcited by 53
  25. 2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
  26. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 41
  27. 2020"You are grounded!": Latent Name Artifacts in Pre-trained Language Modelscited by 39
  28. 2019Linking artificial and human neural representations of languagecited by 37
  29. 2019Sunny and Dark Outside?! Improving Answer Consistency in VQA through Entailed Question Generationcited by 36
  30. 2021Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processingcited by 30
  31. 2021Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue modelscited by 23
  32. 2022Language model acceptability judgements are not always robust to contextcited by 22
  33. 2021Do Question Answering Modeling Improvements Hold Across Benchmarks?cited by 20
  34. 2022A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processingcited by 20
  35. 2023WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Modelscited by 19
  36. 2024Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)cited by 17
  37. 2019Metrology for AI: From Benchmarks to Instrumentscited by 12
  38. 2023The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarkscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025Maestro: Self-Improving Text-to-Image Generation via Agent Orchestrationcited by 18
  2. 2025Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimationcited by 6
  3. 2025DIMCIM: A Quantitative Evaluation Framework for Default-mode Diversity and Generalization in Text-to-Image Generative Modelscited by 4
  4. 2025CORDIAL: Can Multimodal Large Language Models Effectively Understand Coherence Relationships?cited by 3
  5. 2025Multimodal Language Models as Text-to-Image Model Evaluatorscited by 3
  6. 2026SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosiscited by 3
  7. 2026Do Image-Text Metrics Respect Semantic Invariances?cited by 1
  8. 2026Physics Question Scene Graph: Fine-grained Evaluation of Physical Plausibility in Text-to-Video Generationcited by 1
  9. 2025Cost Savings from Automatic Quality Assessment of Generated Imagescited by 0
  10. 2026System-Mediated Attention Imbalances Make Vision-Language Models Say Yescited by 0
  11. 2026Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluationcited by 0