mlsanityData to 7 Oct 2026

DALL-E 2 Fails to Reliably Capture Common Syntactic Processes

arXiv 2210.12889 ↗First posted 23 Oct 2022cs.CL cs.CVcited by 31

Not on the map. Why?

Abstract (arXiv)

Machine intelligence is increasingly being linked to claims about sentience, language processing, and an ability to comprehend and transform natural language into a range of stimuli. We systematically analyze the ability of DALL-E 2 to capture 8 grammatical phenomena pertaining to compositionality that are widely discussed in linguistics and pervasive in human language: binding principles and coreference, passives, word order, coordination, comparatives, negation, ellipsis, and structural ambiguity. Whereas young children routinely master these phenomena, learning systematic mappings between syntax and semantics, DALL-E 2 is unable to reliably infer meanings that are consistent with the syntax. These results challenge recent claims concerning the capacity of such systems to understand of human language. We make available the full set of test materials as a benchmark for future testing.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 11 of its 41 references that are in the map. First posted on arXiv 23 Oct 2022.

Builds on

11 of 41 references.

References · 11

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  3. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 194
  4. 2022A very preliminary analysis of DALL-E 2cited by 82
  5. 2022Meaning without reference in large language modelscited by 76
  6. 2022Testing Relational Understanding in Text-Guided Image Generationcited by 59
  7. 2022DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Modelscited by 50
  8. 2022Discovering the Hidden Vocabulary of DALLE-2cited by 42
  9. 2018Innateness, AlphaZero, and Artificial Intelligencecited by 10
  10. 2022Entailment Semantics Can Be Extracted from an Ideal Language Modelcited by 10
  11. 2022longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attackscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 13; 2024: 12; 2025: 2; 2026: 3.

31 papers.

Sort:

By area:

  1. 2023Teaching CLIP to Count to Tencited by 194
  2. 2023Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignmentcited by 169
  3. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  4. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  5. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  6. 2023Testing AI on language comprehension tasks reveals insensitivity to underlying meaningcited by 45
  7. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  8. 2023I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphorscited by 31
  9. 2023Discovering Failure Modes of Text-guided Diffusion Models via Adversarial Searchcited by 25
  10. 2023Multi-Domain Causal Representation Learning via Weak Distributional Invariancescited by 23
  11. 2023Auditing Gender Presentation Differences in Text-to-Image Modelscited by 22
  12. 2025Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformerscited by 19
19 more
  1. 2023A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the W0rld behind W0rds?cited by 17
  2. 2024Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Referencecited by 15
  3. 2024Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizingcited by 15
  4. 2024Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Modelscited by 14
  5. 2024Evaluating Numerical Reasoning in Text-to-Image Modelscited by 12
  6. 2024Creativity in AI: Progresses and Challengescited by 7
  7. 2024Swing-by Dynamics in Concept Learning and Compositional Generalizationcited by 5
  8. 2023Transferring Visual Attributes from Natural Language to Verified Image Generationcited by 4
  9. 2024A Misleading Gallery of Fluid Motion by Generative Artificial Intelligencecited by 3
  10. 2024A Comparative Investigation of Compositional Syntax and Semantics in DALL-E 2cited by 2
  11. 2024Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsitycited by 2
  12. 2024Infusing Prompts with Syntax and Semanticscited by 2
  13. 2025Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptionscited by 2
  14. 2023A Pilot Evaluation of ChatGPT and DALL-E 2 on Decision Making and Spatial Reasoningcited by 1
  15. 2023The Quo Vadis of the Relationship between Language and Large Language Modelscited by 1
  16. 2026Multilingual Large Language Models do not comprehend all natural languages to equal degreescited by 1
  17. 2026The Art That Poses Back: Assessing AI Pastiches after Contemporary Artworkscited by 1
  18. 2024What is a word?cited by 0
  19. 2026Tracing the ongoing emergence of human-like reasoning in Large Language Modelscited by 0