# Rare Text Semantics Were Always There in Your Diffusion Transformer

arXiv: https://arxiv.org/abs/2510.03886 · HTML: https://mlsanity.com/paper/2510.03886/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 19 of its 54 references that are in the map.
- First posted on arXiv 4 Oct 2025.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 19 of 19)

- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment](https://mlsanity.com/paper/2403.05135/) (2024)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Accelerating Diffusion Transformers with Token-wise Feature Caching](https://mlsanity.com/paper/2410.05317/) (2024)
- [CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models](https://mlsanity.com/paper/2406.08070/) (2024)
- [Stable Flow: Vital Layers for Training-Free Image Editing](https://mlsanity.com/paper/2411.14430/) (2024)
- [ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation](https://mlsanity.com/paper/2406.18522/) (2024)
- [Dynamic Diffusion Transformer](https://mlsanity.com/paper/2410.03456/) (2024)
- [ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer](https://mlsanity.com/paper/2410.00086/) (2024)
- [ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features](https://mlsanity.com/paper/2502.04320/) (2025)
- [Dimensionality compression and expansion in Deep Neural Networks](https://mlsanity.com/paper/1906.00443/) (2019)
- [Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance](https://mlsanity.com/paper/2410.22376/) (2024)
- [Improving Compositional Attribute Binding in Text-to-Image Generative Models via Enhanced Text Embeddings](https://mlsanity.com/paper/2406.07844/) (2024)
- [Improving Long-Text Alignment for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2410.11817/) (2024)
- [End-to-end Training for Text-to-Image Synthesis using Dual-Text Embeddings](https://mlsanity.com/paper/2502.01507/) (2025)

## Built on by (most cited first, 4 of 4)

- [Physics in 2-Steps: Locking Motion Priors Before Visual Refinement Erases Them](https://mlsanity.com/paper/2606.06361/) (2026)
- [Pseudo-Unification: Entropy Probing Reveals Divergent Information Patterns in Unified Multimodal Models](https://mlsanity.com/paper/2604.10949/) (2026)
- [When Do Diffusion Models learn to Generate Multiple Objects?](https://mlsanity.com/paper/2605.00273/) (2026)
- [RAIGen: Rare Attribute Identification in Text-to-Image Generative Models](https://mlsanity.com/paper/2602.06806/) (2026)

## Abstract (arXiv)

Starting from flow- and diffusion-based transformers, Multi-modal Diffusion Transformers (MM-DiTs) have reshaped text-to-vision generation, gaining acclaim for exceptional visual fidelity. As these models advance, users continually push the boundary with imaginative or rare prompts, which advanced models still falter in generating, since their concepts are often too scarce to leave a strong imprint during pre-training. In this paper, we propose a simple yet effective intervention that surfaces rare semantics inside MM-DiTs without additional training steps, data, denoising-time optimization, or reliance on external modules (e.g., large language models). In particular, the joint-attention mechanism intrinsic to MM-DiT sequentially updates text embeddings alongside image embeddings throughout transformer blocks. We find that by mathematically expanding representational basins around text token embeddings via variance scale-up before the joint-attention blocks, rare semantics clearly emerge in MM-DiT's outputs. Furthermore, our results generalize effectively across text-to-vision tasks, including text-to-image, text-to-video, and text-driven image editing. Our work invites generative models to reveal the semantics that users intend, once hidden yet ready to surface.