mlsanityData to 7 Oct 2026

TeTIm-Eval: a novel curated evaluation data set for comparing text-to-image models

arXiv 2212.07839 ↗First posted 15 Dec 2022cs.CV cs.CL cs.LGcited by 0

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Evaluating and comparing text-to-image models is a challenging problem. Significant advances in the field have recently been made, piquing interest of various industrial sectors. As a consequence, a gold standard in the field should cover a variety of tasks and application contexts. In this paper a novel evaluation approach is experimented, on the basis of: (i) a curated data set, made by high-quality royalty-free image-text pairs, divided into ten categories; (ii) a quantitative metric, the CLIP-score, (iii) a human evaluation task to distinguish, for a given text, the real and the generated images. The proposed method has been applied to the most recent models, i.e., DALLE2, Latent Diffusion, Stable Diffusion, GLIDE and Craiyon. Early experimental results show that the accuracy of the human judgement is fully coherent with the CLIP-score. The dataset has been made available to the public.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 26 of its 35 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

26 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
11 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  3. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  4. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  5. 2016Generative Adversarial Text to Image Synthesiscited by 735
  6. 2021A Survey of Transformerscited by 399
  7. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  8. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  9. 2019Effectively Unbiased FID and Inception Score and where to find themcited by 161
  10. 2021Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Searchcited by 57
  11. 2021Adversarial Text-to-Image Synthesis: A Reviewcited by 52

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models
  2. 2022Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark
  3. 2025Multimodal Language Models as Text-to-Image Model Evaluators
  4. 2024Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark
  5. 2024FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models
  6. 2025Evaluating the Evaluators: Metrics for Compositional Text-to-Image Generation
  7. 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models
  8. 2024Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models