mlsanityData to 14 Oct 2026

Testing Relational Understanding in Text-Guided Image Generation

arXiv 2208.00005 ↗First posted 29 Jul 2022cs.CV cs.AI cs.LGcited by 59

Not on the map. Why?

Abstract (arXiv)

Relations are basic building blocks of human cognition. Classic and recent work suggests that many relations are early developing, and quickly perceived. Machine models that aspire to human-level perception and reasoning should reflect the ability to recognize and reason generatively about relations. We report a systematic empirical examination of a recent text-guided image generation model (DALL-E 2), using a set of 15 basic physical and social relations studied or proposed in the literature, and judgements from human participants (N = 169). Overall, we find that only ~22% of images matched basic relation prompts. Based on a quantitative examination of people's judgments, we suggest that current image generation models do not yet have a grasp of even basic relations involving simple objects and agents. We examine reasons for model successes and failures, and suggest possible improvements based on computations observed in biological intelligence.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Builds on 13 of its 41 references that are in the map. First posted on arXiv 29 Jul 2022.

Builds on

13 of 41 references.

References · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  7. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  8. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  9. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 306
  10. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  11. 2022A very preliminary analysis of DALL-E 2cited by 82
  12. 2022Perspective (In)consistency of Paint by Textcited by 34
1 more
  1. 2020RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spacescited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 16; 2024: 23; 2025: 5; 2026: 5.

59 papers.

Sort:

By area:

  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  2. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  3. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  4. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignmentcited by 171
  5. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  6. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  7. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 119
  8. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 106
  9. 2023Consciousness in Artificial Intelligence: Insights from the Science of Consciousnesscited by 104
  10. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 97
  11. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 93
  12. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
38 more
  1. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problemcited by 75
  2. 2022DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Modelscited by 51
  3. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  4. 2023Interpretable Diffusion via Information Decompositioncited by 37
  5. 2023Mass-Producing Failures of Multimodal Systems with Language Modelscited by 36
  6. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 35
  7. 2022DALL-E 2 Fails to Reliably Capture Common Syntactic Processescited by 31
  8. 2022Exploiting Cultural Biases via Homoglyphs in Text-to-Image Synthesiscited by 29
  9. 2023Compositional Sculpting of Iterative Generative Processescited by 27
  10. 2023Auditing Gender Presentation Differences in Text-to-Image Modelscited by 22
  11. 2023CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decodingcited by 20
  12. 2024Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Modelscited by 18
  13. 2023MEWL: Few-shot multimodal word learning with referential uncertaintycited by 17
  14. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 17
  15. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decompositioncited by 16
  16. 2024Text Guided Image Editing with Automatic Concept Locating and Forgettingcited by 12
  17. 2024Discriminative Probing and Tuning for Text-to-Image Generationcited by 11
  18. 2024Revealing Vision-Language Integration in the Brain with Multimodal Networkscited by 11
  19. 2023Graph Neural Networks in Vision-Language Image Understanding: A Surveycited by 10
  20. 2024Relations, Negations, and Numbers: Looking for Logic in Generative Text-to-Image Modelscited by 9
  21. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  22. 2024Information Theoretic Text-to-Image Alignmentcited by 8
  23. 2022What Does DALL-E 2 Know About Radiology?cited by 7
  24. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  25. 2024Creativity in AI: Progresses and Challengescited by 7
  26. 2024Evaluating the Generation of Spatial Relations in Text and Image Generative Modelscited by 6
  27. 2024Swing-by Dynamics in Concept Learning and Compositional Generalizationcited by 5
  28. 2025Evaluating Compositional Scene Understanding in Multimodal Generative Modelscited by 5
  29. 2024Interpretable Measures of Conceptual Similarity by Complexity-Constrained Descriptive Auto-Encodingcited by 4
  30. 2022Language Models Understand Us, Poorlycited by 3
  31. 2023STUPD: A Synthetic Dataset for Spatial and Temporal Relation Reasoningcited by 3
  32. 2024Overcoming classic challenges for artificial neural networks by providing incentives and practicecited by 3
  33. 2025SynC: Synthetic Image Caption Dataset Refinement with One-to-many Mapping for Zero-shot Image Captioningcited by 3
  34. 2026Circuit Mechanisms for Spatial Relation Generation in Diffusion Transformerscited by 3
  35. 2024CUPID: Contextual Understanding of Prompt-conditioned Image Distributionscited by 2
  36. 2025Computer Vision Models Show Human-Like Sensitivity to Geometric and Topological Conceptscited by 2
  37. 2024Bridging the Intent Gap: Knowledge-Enhanced Visual Generationcited by 1
  38. 2024Classification-Denoising Networkscited by 1

50 most cited of 59 All 59 (JSON)