# DALL-E 2 Fails to Reliably Capture Common Syntactic Processes

arXiv: https://arxiv.org/abs/2210.12889 · HTML: https://mlsanity.com/paper/2210.12889/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 41 references that are in the map.
- First posted on arXiv 23 Oct 2022.

## Builds on (most cited first, 11 of 11)

- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence](https://mlsanity.com/paper/2002.06177/) (2020)
- [A very preliminary analysis of DALL-E 2](https://mlsanity.com/paper/2204.13807/) (2022)
- [Meaning without reference in large language models](https://mlsanity.com/paper/2208.02957/) (2022)
- [Testing Relational Understanding in Text-Guided Image Generation](https://mlsanity.com/paper/2208.00005/) (2022)
- [DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Models](https://mlsanity.com/paper/2210.10606/) (2022)
- [Discovering the Hidden Vocabulary of DALLE-2](https://mlsanity.com/paper/2206.00169/) (2022)
- [Innateness, AlphaZero, and Artificial Intelligence](https://mlsanity.com/paper/1801.05667/) (2018)
- [Entailment Semantics Can Be Extracted from an Ideal Language Model](https://mlsanity.com/paper/2209.12407/) (2022)
- [longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks](https://mlsanity.com/paper/2206.14729/) (2022)

## Built on by (most cited first, 30 of 31)

- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment](https://mlsanity.com/paper/2306.08877/) (2023)
- [Holistic Evaluation of Text-To-Image Models](https://mlsanity.com/paper/2311.04287/) (2023)
- [Benchmarking Spatial Relationships in Text-to-Image Generation](https://mlsanity.com/paper/2212.10015/) (2022)
- [Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task](https://mlsanity.com/paper/2310.09336/) (2023)
- [Testing AI on language comprehension tasks reveals insensitivity to underlying meaning](https://mlsanity.com/paper/2302.12313/) (2023)
- [Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space](https://mlsanity.com/paper/2406.19370/) (2024)
- [I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors](https://mlsanity.com/paper/2305.14724/) (2023)
- [Discovering Failure Modes of Text-guided Diffusion Models via Adversarial Search](https://mlsanity.com/paper/2306.00974/) (2023)
- [Multi-Domain Causal Representation Learning via Weak Distributional Invariances](https://mlsanity.com/paper/2310.02854/) (2023)
- [Auditing Gender Presentation Differences in Text-to-Image Models](https://mlsanity.com/paper/2302.03675/) (2023)
- [Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers](https://mlsanity.com/paper/2501.08537/) (2025)
- [A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the W0rld behind W0rds?](https://mlsanity.com/paper/2308.00109/) (2023)
- [Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing](https://mlsanity.com/paper/2405.05409/) (2024)
- [Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Reference](https://mlsanity.com/paper/2404.14883/) (2024)
- [Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models](https://mlsanity.com/paper/2408.06518/) (2024)
- [Evaluating Numerical Reasoning in Text-to-Image Models](https://mlsanity.com/paper/2406.14774/) (2024)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [Swing-by Dynamics in Concept Learning and Compositional Generalization](https://mlsanity.com/paper/2410.08309/) (2024)
- [Transferring Visual Attributes from Natural Language to Verified Image Generation](https://mlsanity.com/paper/2305.15026/) (2023)
- [A Misleading Gallery of Fluid Motion by Generative Artificial Intelligence](https://mlsanity.com/paper/2405.15406/) (2024)
- [A Comparative Investigation of Compositional Syntax and Semantics in DALL-E 2](https://mlsanity.com/paper/2403.12294/) (2024)
- [Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity](https://mlsanity.com/paper/2410.13964/) (2024)
- [Infusing Prompts with Syntax and Semantics](https://mlsanity.com/paper/2412.06107/) (2024)
- [Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions](https://mlsanity.com/paper/2510.16540/) (2025)
- [A Pilot Evaluation of ChatGPT and DALL-E 2 on Decision Making and Spatial Reasoning](https://mlsanity.com/paper/2302.09068/) (2023)
- [The Quo Vadis of the Relationship between Language and Large Language Models](https://mlsanity.com/paper/2310.11146/) (2023)
- [Multilingual Large Language Models do not comprehend all natural languages to equal degrees](https://mlsanity.com/paper/2602.20065/) (2026)
- [The Art That Poses Back: Assessing AI Pastiches after Contemporary Artworks](https://mlsanity.com/paper/2603.06324/) (2026)
- [What is a word?](https://mlsanity.com/paper/2402.12605/) (2024)

## Abstract (arXiv)

Machine intelligence is increasingly being linked to claims about sentience, language processing, and an ability to comprehend and transform natural language into a range of stimuli. We systematically analyze the ability of DALL-E 2 to capture 8 grammatical phenomena pertaining to compositionality that are widely discussed in linguistics and pervasive in human language: binding principles and coreference, passives, word order, coordination, comparatives, negation, ellipsis, and structural ambiguity. Whereas young children routinely master these phenomena, learning systematic mappings between syntax and semantics, DALL-E 2 is unable to reliably infer meanings that are consistent with the syntax. These results challenge recent claims concerning the capacity of such systems to understand of human language. We make available the full set of test materials as a benchmark for future testing.