# A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers

arXiv: https://arxiv.org/abs/2602.21596 · HTML: https://mlsanity.com/paper/2602.21596/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 25 Feb 2026.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 33)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers](https://mlsanity.com/paper/2401.08740/) (2024)
- [Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think](https://mlsanity.com/paper/2410.06940/) (2024)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models](https://mlsanity.com/paper/2501.01423/) (2025)
- [MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer](https://mlsanity.com/paper/2303.14389/) (2023)
- [The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers](https://mlsanity.com/paper/2210.06313/) (2022)
- [How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learning](https://mlsanity.com/paper/2203.16262/) (2022)
- [Cascade RPN: Delving into High-Quality Region Proposal Network with Adaptive Convolution](https://mlsanity.com/paper/1909.06720/) (2019)
- [Modality Shifting Attention Network for Multi-modal Video Question Answering](https://mlsanity.com/paper/2007.02036/) (2020)
- [Diffusion Models without Classifier-free Guidance](https://mlsanity.com/paper/2502.12154/) (2025)
- [FlexiEdit: Frequency-Aware Latent Refinement for Enhanced Non-Rigid Editing](https://mlsanity.com/paper/2407.17850/) (2024)
- [Unified Continuous Generative Models](https://mlsanity.com/paper/2505.07447/) (2025)
- [Cross-view Masked Diffusion Transformers for Person Image Synthesis](https://mlsanity.com/paper/2402.01516/) (2024)
- [On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learning](https://mlsanity.com/paper/2208.05744/) (2022)
- [ITA-MDT: Image-Timestep-Adaptive Masked Diffusion Transformer Framework for Image-Based Virtual Try-On](https://mlsanity.com/paper/2503.20418/) (2025)
- [MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation](https://mlsanity.com/paper/2410.02130/) (2024)
- [LAD: A Hybrid Deep Learning System for Benign Paroxysmal Positional Vertigo Disorders Diagnostic](https://mlsanity.com/paper/2210.08282/) (2022)
- [FlowDrag: 3D-aware Drag-based Image Editing with Mesh-guided Deformation Vector Flow Fields](https://mlsanity.com/paper/2507.08285/) (2025)
- [Self-Supervised Visual Representation Learning via Residual Momentum](https://mlsanity.com/paper/2211.09861/) (2022)

## Built on by (most cited first, 2 of 2)

- [Let It Be Simple: One-Step Action Generation for Vision-Language-Action Models](https://mlsanity.com/paper/2606.05737/) (2026)
- [Few Channels Draw The Whole Picture: Revealing Massive Activations in Diffusion Transformers](https://mlsanity.com/paper/2605.13974/) (2026)

## Abstract (arXiv)

Diffusion Transformers have achieved state-of-the-art performance in class-conditional and multimodal generation, yet the structure of their learned conditional embeddings remains poorly understood. In this work, we present the first systematic study of these embeddings and uncover a notable redundancy: class-conditioned embeddings exhibit extreme angular similarity, exceeding 99% on ImageNet-1K, while continuous-condition tasks such as pose-guided image generation and video-to-audio generation reach over 99.9%. We further find that semantic information is concentrated in a small subset of dimensions, with head dimensions carrying the dominant signal and tail dimensions contributing minimally. By pruning low-magnitude dimensions--removing up to two-thirds of the embedding space--we show that generation quality and fidelity remain largely unaffected, and in some cases improve. These results reveal a semantic bottleneck in Transformer-based diffusion models, providing new insights into how semantics are encoded and suggesting opportunities for more efficient conditioning mechanisms.