mlsanityData to 7 Oct 2026

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models

arXiv 2202.04053 ↗First posted 8 Feb 2022cs.CV cs.AI cs.CLcited by 230

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Recently, DALL-E, a multimodal transformer language model, and its variants, including diffusion models, have shown high-quality text-to-image generation capabilities. However, despite the realistic image generation results, there has not been a detailed analysis of how to evaluate such models. In this work, we investigate the visual reasoning capabilities and social biases of different text-to-image models, covering both multimodal transformer language models and diffusion models. First, we measure three visual reasoning skills: object recognition, object counting, and spatial relation understanding. For this, we propose PaintSkills, a compositional diagnostic evaluation dataset that measures these skills. Despite the high-fidelity image generation capability, a large gap exists between the performance of recent models and the upper bound accuracy in object counting and spatial relation understanding skills. Second, we assess the gender and skin tone biases by measuring the gender/skin tone distribution of generated images across various professions and attributes. We demonstrate that recent text-to-image generation models learn specific biases about gender and skin tone from web image-text pairs. We hope our work will help guide future progress in improving text-to-image generation models on visual reasoning skills and learning socially unbiased representations. Code and data: https://github.com/j-min/DallEval

Key facts. Cited by 230 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 56 of its 105 references that are in the map. First posted on arXiv 8 Feb 2022.

Builds on

56 of 105 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2016Improved Techniques for Training GANscited by 2,938
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  11. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  13. 2018Hierarchical Neural Story Generationcited by 1,182
  14. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  15. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  16. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  17. 2016Generative Adversarial Text to Image Synthesiscited by 735
  18. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  19. 2017How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)cited by 524
  20. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  21. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  22. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  23. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  24. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  25. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  26. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 279
  27. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  28. 2015Generating Images from Captions with Attentioncited by 135
  29. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  30. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  31. 2020Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biasescited by 107
  32. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  33. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  34. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  35. 2020Measuring Social Biases in Grounded Vision and Language Embeddingscited by 61
  36. 2020X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformerscited by 58
  37. 2021Adversarial Text-to-Image Synthesis: A Reviewcited by 52
  38. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 27; 2024: 99; 2025: 66; 2026: 37.

230 papers.

Sort:

By area:

  1. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  2. 2023Unified Concept Editing in Diffusion Modelscited by 377
  3. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  4. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  5. 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
  6. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  7. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
  8. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  9. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  10. 2023Social Biases through the Text-to-Image Generation Lenscited by 126
  11. 2023Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluationcited by 108
  12. 2023LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluationcited by 85
38 more
  1. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  2. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  3. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  4. 2023The Bias Amplification Paradox in Text-to-Image Generationcited by 69
  5. 2025KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Modelscited by 69
  6. 2024Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigationcited by 66
  7. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  8. 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
  9. 2023T3Bench: Benchmarking Current Progress in Text-to-3D Generationcited by 56
  10. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
  11. 2024Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?cited by 52
  12. 2023TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Modelscited by 51
  13. 2024LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Modelscited by 49
  14. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  15. 2024Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMscited by 48
  16. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  17. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
  18. 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
  19. 2024DiP-GO: A Diffusion Pruner via Few-step Gradient Optimizationcited by 43
  20. 2025The Amazon Nova Family of Models: Technical Report and Model Cardcited by 41
  21. 2023TokenCompose: Text-to-Image Diffusion with Token-level Supervisioncited by 40
  22. 2023COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairscited by 39
  23. 2024BitsFusion: 1.99 bits Weight Quantization of Diffusion Modelcited by 37
  24. 2024An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevancecited by 36
  25. 2024Consistency-diversity-realism Pareto fronts of conditional image generative modelscited by 35
  26. 2024FairRAG: Fair Human Generation via Fair Retrieval Augmentationcited by 34
  27. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  28. 2024Getting it Right: Improving Spatial Consistency in Text-to-Image Modelscited by 30
  29. 2024Beyond Aesthetics: Cultural Competence in Text-to-Image Modelscited by 30
  30. 2023DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planningcited by 28
  31. 2024ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generationcited by 27
  32. 2024Text2LiDAR: Text-guided LiDAR Point Cloud Generation via Equirectangular Transformercited by 26
  33. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  34. 2023Stable Diffusion Exposed: Gender Bias from Prompt to Imagecited by 25
  35. 2025ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editingcited by 23
  36. 2022Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Surveycited by 22
  37. 2023Auditing Gender Presentation Differences in Text-to-Image Modelscited by 22
  38. 2024Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusioncited by 22

50 most cited of 230 All 230 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment
  2. 2025MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning
  3. 2025ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation
  4. 2022Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark
  5. 2025Thinking with Generated Images
  6. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  7. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals
  8. 2026Evaluating Reasoning Fidelity in Visual Text Generation