# DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Models

arXiv: https://arxiv.org/abs/2210.10606 · HTML: https://mlsanity.com/paper/2210.10606/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 11 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 6 of 6)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A very preliminary analysis of DALL-E 2](https://mlsanity.com/paper/2204.13807/) (2022)
- [Testing Relational Understanding in Text-Guided Image Generation](https://mlsanity.com/paper/2208.00005/) (2022)
- [Underspecification in Scene Description-to-Depiction Tasks](https://mlsanity.com/paper/2210.05815/) (2022)

## Built on by (most cited first, 30 of 51)

- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [Linguistic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment](https://mlsanity.com/paper/2306.08877/) (2023)
- [DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion](https://mlsanity.com/paper/2304.06025/) (2023)
- [What You See is What You Read? Improving Text-Image Alignment Evaluation](https://mlsanity.com/paper/2305.10400/) (2023)
- [Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task](https://mlsanity.com/paper/2310.09336/) (2023)
- [DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback](https://mlsanity.com/paper/2311.17946/) (2023)
- [Make It Count: Text-to-Image Generation with an Accurate Number of Objects](https://mlsanity.com/paper/2406.10210/) (2024)
- [Testing AI on language comprehension tasks reveals insensitivity to underlying meaning](https://mlsanity.com/paper/2302.12313/) (2023)
- [Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space](https://mlsanity.com/paper/2406.19370/) (2024)
- [The Hidden Language of Diffusion Models](https://mlsanity.com/paper/2306.00966/) (2023)
- [DALL-E 2 Fails to Reliably Capture Common Syntactic Processes](https://mlsanity.com/paper/2210.12889/) (2022)
- [When are Lemons Purple? The Concept Association Bias of Vision-Language Models](https://mlsanity.com/paper/2212.12043/) (2022)
- [Auditing Gender Presentation Differences in Text-to-Image Models](https://mlsanity.com/paper/2302.03675/) (2023)
- [Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignment](https://mlsanity.com/paper/2312.03766/) (2023)
- [Object-Conditioned Energy-Based Attention Map Alignment in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2404.07389/) (2024)
- [DiffusionPID: Interpreting Diffusion via Partial Information Decomposition](https://mlsanity.com/paper/2406.05191/) (2024)
- [Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Reference](https://mlsanity.com/paper/2404.14883/) (2024)
- [Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models](https://mlsanity.com/paper/2408.06518/) (2024)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [Text Guided Image Editing with Automatic Concept Locating and Forgetting](https://mlsanity.com/paper/2405.19708/) (2024)
- [Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control](https://mlsanity.com/paper/2404.13766/) (2024)
- [Image Generation from Contextually-Contradictory Prompts](https://mlsanity.com/paper/2506.01929/) (2025)
- [Schr"odinger's Bat: Diffusion Models Sometimes Generate Polysemous Words in Superposition](https://mlsanity.com/paper/2211.13095/) (2022)
- [Global-Local Image Perceptual Score (GLIPS): Evaluating Photorealistic Quality of AI-Generated Images](https://mlsanity.com/paper/2405.09426/) (2024)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [Inverse Painting: Reconstructing The Painting Process](https://mlsanity.com/paper/2409.20556/) (2024)
- [Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information](https://mlsanity.com/paper/2305.01788/) (2023)
- [Total Selfie: Generating Full-Body Selfies](https://mlsanity.com/paper/2308.14740/) (2023)
- [Swing-by Dynamics in Concept Learning and Compositional Generalization](https://mlsanity.com/paper/2410.08309/) (2024)
- [GRADE: Quantifying Sample Diversity in Text-to-Image Models](https://mlsanity.com/paper/2410.22592/) (2024)

## Abstract (arXiv)

We study the way DALLE-2 maps symbols (words) in the prompt to their references (entities or properties of entities in the generated image). We show that in stark contrast to the way human process language, DALLE-2 does not follow the constraint that each word has a single role in the interpretation, and sometimes re-use the same symbol for different purposes. We collect a set of stimuli that reflect the phenomenon: we show that DALLE-2 depicts both senses of nouns with multiple senses at once; and that a given word can modify the properties of two distinct entities in the image, or can be depicted as one object and also modify the properties of another object, creating a semantic leakage of properties between entities. Taken together, our study highlights the differences between DALLE-2 and human language processing and opens an avenue for future study on the inductive biases of text-to-image models.