# Text-to-Image Models Need Less from Text Encoders Than You Think

arXiv: https://arxiv.org/abs/2606.03715 · HTML: https://mlsanity.com/paper/2606.03715/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment](https://mlsanity.com/paper/2403.05135/) (2024)
- [SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality](https://mlsanity.com/paper/2306.14610/) (2023)
- [What the DAAM: Interpreting Stable Diffusion Using Cross Attention](https://mlsanity.com/paper/2210.04885/) (2022)
- [DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation](https://mlsanity.com/paper/2406.16855/) (2024)
- [Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation](https://mlsanity.com/paper/2401.06591/) (2024)
- [VHELM: A Holistic Evaluation of Vision Language Models](https://mlsanity.com/paper/2410.07112/) (2024)
- [Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space](https://mlsanity.com/paper/2406.19370/) (2024)
- [Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelines](https://mlsanity.com/paper/2403.05846/) (2024)
- [Scaling Down Text Encoders of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2503.19897/) (2025)
- [Circuit Mechanisms for Spatial Relation Generation in Diffusion Transformers](https://mlsanity.com/paper/2601.06338/) (2026)
- [DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Models](https://mlsanity.com/paper/2510.15015/) (2025)

## Built on by (most cited first, 1 of 1)

- [The Plan, Not the Decoder: Diagnosing and Repairing Compositional Failure in Reasoning-Augmented Text-to-Image Generation](https://mlsanity.com/paper/2608.21713/) (2026)

## Abstract (arXiv)

Text-to-image models rely on text prompts as their primary interface to human intent. Prompts are encoded by a text encoder into embeddings that condition the image generation process. Beyond individual token meanings, text embeddings encode contextual information across the full prompt, such as compositionality and attribute binding. However, whether image models actually exploit this richer information remains underexplored. Here, we address the question: Which aspects of text representation are essential for image generation? We show that text-to-image diffusion transformer-based models commonly rely only on two relatively straightforward aspects of text representations: (i) the merging of adjacent tokens into a word representation, for words spanning multiple tokens, and (ii) word order, which is imprinted by the positional embedding of the text-encoder. To show this, we construct a new text embedding that encodes only individual word meanings and order but lacks any contextual information about the full prompt. We find that this bag of position-tagged words representation is sufficient to successfully guide image generation, achieving visual quality and text fidelity that are on par with full text embedding-guided generation. This demonstrates that, contrary to common belief, text-to-image models often do not use the rich information encoded in the text embedding beyond individual word meanings and word order. Instead, the decoding of complex linguistic structures is performed by the image model itself. Project webpage: https://nsping13.github.io/contextless-TTI/