mlsanityData to 7 Oct 2026

A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers

arXiv 2602.21596 ↗First posted 25 Feb 2026cs.CVcited by 2

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Diffusion Transformers have achieved state-of-the-art performance in class-conditional and multimodal generation, yet the structure of their learned conditional embeddings remains poorly understood. In this work, we present the first systematic study of these embeddings and uncover a notable redundancy: class-conditioned embeddings exhibit extreme angular similarity, exceeding 99% on ImageNet-1K, while continuous-condition tasks such as pose-guided image generation and video-to-audio generation reach over 99.9%. We further find that semantic information is concentrated in a small subset of dimensions, with head dimensions carrying the dominant signal and tail dimensions contributing minimally. By pruning low-magnitude dimensions--removing up to two-thirds of the embedding space--we show that generation quality and fidelity remain largely unaffected, and in some cases improve. These results reveal a semantic bottleneck in Transformer-based diffusion models, providing new insights into how semantics are encoded and suggesting opportunities for more efficient conditioning mechanisms.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 33 of its 41 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

33 of 41 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Unified Continuous Generative Models12 May
  2. 2025Diffusion Models without Classifier-free Guidance17 Feb
  3. 2025E-MD3C: Taming Masked Diffusion Transformers for Efficient Zero-Shot Object Customization13 Feb
  4. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models2 Jan
  5. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think9 Oct
  6. 2024Cross-view Masked Diffusion Transformers for Person Image Synthesis2 Feb
  7. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers16 Jan
  8. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizer25 Mar
  9. 2022Scalable Diffusion Models with Transformers19 Dec
  10. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  11. 2021Diffusion Models Beat GANs on Image Synthesis11 May

Other references · 22

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  5. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  8. 2017SphereFace: Deep Hypersphere Embedding for Face Recognitioncited by 733
  9. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  10. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  11. 2022How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learningcited by 65
  12. 2019Cascade RPN: Delving into High-Quality Region Proposal Network with Adaptive Convolutioncited by 57
10 more
  1. 2020Modality Shifting Attention Network for Multi-modal Video Question Answeringcited by 38
  2. 2024FlexiEdit: Frequency-Aware Latent Refinement for Enhanced Non-Rigid Editingcited by 31
  3. 2022On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learningcited by 10
  4. 2025ITA-MDT: Image-Timestep-Adaptive Masked Diffusion Transformer Framework for Image-Based Virtual Try-Oncited by 10
  5. 2024MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generationcited by 9
  6. 2022LAD: A Hybrid Deep Learning System for Benign Paroxysmal Positional Vertigo Disorders Diagnosticcited by 8
  7. 2025FlowDrag: 3D-aware Drag-based Image Editing with Mesh-guided Deformation Vector Flow Fieldscited by 8
  8. 2022Self-Supervised Visual Representation Learning via Residual Momentumcited by 7
  9. 2021Self-supervised Learning with Local Attention-Aware Featurecited by 5
  10. 2025Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Let It Be Simple: One-Step Action Generation for Vision-Language-Action Modelscited by 3
  2. 2026Few Channels Draw The Whole Picture: Revealing Massive Activations in Diffusion Transformerscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Representation by Design in Generation: Cross-View Class-Token Alignment in Diffusion Transformers
  2. 2024Diffusion Models Need Visual Priors for Image Generation
  3. 2025Automated Learning of Semantic Embedding Representations for Diffusion Models
  4. 2023Visual Chain-of-Thought Diffusion Models
  5. 2026Training-Free Representation Guidance for Diffusion Models with a Representation Alignment Projector
  6. 2024On Improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models
  7. 2025Rare Text Semantics Were Always There in Your Diffusion Transformer
  8. 2025Diffusion Transformers with Representation Autoencoders