# Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement

arXiv: https://arxiv.org/abs/2402.09712 · HTML: https://mlsanity.com/paper/2402.09712/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 34 of its 44 references that are in the map.
- First posted on arXiv 15 Feb 2024.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 34)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Diffusion Models: A Comprehensive Survey of Methods and Applications](https://mlsanity.com/paper/2209.00796/) (2022)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Unsupervised Discovery of Interpretable Directions in the GAN Latent Space](https://mlsanity.com/paper/2002.03754/) (2020)
- [Variational Inference of Disentangled Latent Concepts from Unlabeled Observations](https://mlsanity.com/paper/1711.00848/) (2017)
- [InfoVAE: Information Maximizing Variational Autoencoders](https://mlsanity.com/paper/1706.02262/) (2017)
- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset](https://mlsanity.com/paper/1906.03292/) (2019)
- [Object-Centric Slot Diffusion](https://mlsanity.com/paper/2303.10834/) (2023)
- [SlotDiffusion: Object-Centric Generative Modeling with Diffusion Models](https://mlsanity.com/paper/2305.11281/) (2023)
- [SODA: Bottleneck Diffusion Models for Representation Learning](https://mlsanity.com/paper/2311.17901/) (2023)
- [Unsupervised Learning of Compositional Energy Concepts](https://mlsanity.com/paper/2111.03042/) (2021)
- [Unsupervised Representation Learning from Pre-trained Diffusion Probabilistic Models](https://mlsanity.com/paper/2212.12990/) (2022)
- [InfoDiffusion: Representation Learning Using Information Maximizing Diffusion Models](https://mlsanity.com/paper/2306.08757/) (2023)
- [Learning Disentangled Representation by Exploiting Pretrained Generative Models: A Contrastive Learning View](https://mlsanity.com/paper/2102.10543/) (2021)
- [Leveraging sparse and shared feature activations for disentangled representation learning](https://mlsanity.com/paper/2304.07939/) (2023)
- [DisDiff: Unsupervised Disentanglement of Diffusion Probabilistic Models](https://mlsanity.com/paper/2301.13721/) (2023)
- [Where and What? Examining Interpretable Disentangled Representations](https://mlsanity.com/paper/2104.05622/) (2021)
- [Towards Building A Group-based Unsupervised Representation Disentanglement Framework](https://mlsanity.com/paper/2102.10303/) (2021)
- [Learning Representations by Maximizing Mutual Information in Variational Autoencoders](https://mlsanity.com/paper/1912.13361/) (2019)

## Built on by (most cited first, 27 of 27)

- [DiP: Taming Diffusion Models in Pixel Space](https://mlsanity.com/paper/2511.18822/) (2025)
- [Learning Semantic Latent Directions for Accurate and Controllable Human Motion Prediction](https://mlsanity.com/paper/2407.11494/) (2024)
- [SLICE: Semantic Latent Injection via Compartmentalized Embedding for Image Watermarking](https://mlsanity.com/paper/2603.12749/) (2026)
- [Understanding Implosion in Text-to-Image Generative Models](https://mlsanity.com/paper/2409.12314/) (2024)
- [Disentangling Disentangled Representations: Towards Improved Latent Units via Diffusion Models](https://mlsanity.com/paper/2410.23820/) (2024)
- [Causal-Adapter: Taming Text-to-Image Diffusion for Faithful Counterfactual Generation](https://mlsanity.com/paper/2509.24798/) (2025)
- [Exploring Position Encoding in Diffusion U-Net for Training-free High-resolution Image Generation](https://mlsanity.com/paper/2503.09830/) (2025)
- [Harnessing Text-to-Image Diffusion Models for Point Cloud Self-Supervised Learning](https://mlsanity.com/paper/2507.09102/) (2025)
- [Causality-guided Prompt Learning for Vision-language Models via Visual Granulation](https://mlsanity.com/paper/2509.03803/) (2025)
- [Disentangled Representation Learning via Flow Matching](https://mlsanity.com/paper/2602.05214/) (2026)
- [StrADiff: A Structured Source-Wise Adaptive Diffusion Framework for Linear and Nonlinear Blind Source Separation](https://mlsanity.com/paper/2604.04973/) (2026)
- [DiViD: Disentangled Video Diffusion for Static-Dynamic Factorization](https://mlsanity.com/paper/2507.13934/) (2025)
- [Coffee: Controllable Diffusion Fine-tuning](https://mlsanity.com/paper/2511.14113/) (2025)
- [MeDUET: Disentangled Unified Pretraining for 3D Medical Image Synthesis and Analysis](https://mlsanity.com/paper/2602.17901/) (2026)
- [Diff-CA: Separating Common and Salient Factors with Diffusion Models](https://mlsanity.com/paper/2606.06120/) (2026)
- [Dual-Branch Cross-Projection Debiasing through Diffusion-based Disentanglement](https://mlsanity.com/paper/2606.24161/) (2026)
- [Revisiting Diffusion Autoencoder Training for Image Reconstruction Quality](https://mlsanity.com/paper/2504.21368/) (2025)
- [The 1st International Workshop on Disentangled Representation Learning for Controllable Generation (DRL4Real): Methods and Results](https://mlsanity.com/paper/2509.10463/) (2025)
- [Cross-attention Secretly Performs Orthogonal Alignment in Recommendation Models](https://mlsanity.com/paper/2510.09435/) (2025)
- [CLOAK: Contrastive Guidance for Latent Diffusion-Based Data Obfuscation](https://mlsanity.com/paper/2512.12086/) (2025)
- [Disentangled representations via score-based variational autoencoders](https://mlsanity.com/paper/2512.17127/) (2025)
- [XFACTORS: Disentangled Information Bottleneck via Contrastive Supervision](https://mlsanity.com/paper/2601.21688/) (2026)
- [Diverse Dictionary Learning](https://mlsanity.com/paper/2604.17568/) (2026)
- [Awakening the Hydra: Stabilizing Multi-Concept Backdoor Injection in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2605.19698/) (2026)
- [Reflection Separation from a Single Image via Joint Latent Diffusion](https://mlsanity.com/paper/2606.04107/) (2026)
- [CenSynCMB: Centre Maps and Physics-Guided Synthesis for Microbleed Detection](https://mlsanity.com/paper/2607.05325/) (2026)
- [Precipitation Downscaling Using Foundation Model-Conditioned Diffusion](https://mlsanity.com/paper/2608.25858/) (2026)

## Abstract (arXiv)

Disentangled representation learning strives to extract the intrinsic factors within observed data. Factorizing these representations in an unsupervised manner is notably challenging and usually requires tailored loss functions or specific structural designs. In this paper, we introduce a new perspective and framework, demonstrating that diffusion models with cross-attention can serve as a powerful inductive bias to facilitate the learning of disentangled representations. We propose to encode an image to a set of concept tokens and treat them as the condition of the latent diffusion for image reconstruction, where cross-attention over the concept tokens is used to bridge the interaction between the encoder and diffusion. Without any additional regularization, this framework achieves superior disentanglement performance on the benchmark datasets, surpassing all previous methods with intricate designs. We have conducted comprehensive ablation studies and visualization analysis, shedding light on the functioning of this model. This is the first work to reveal the potent disentanglement capability of diffusion models with cross-attention, requiring no complex designs. We anticipate that our findings will inspire more investigation on exploring diffusion for disentangled representation learning towards more sophisticated data analysis and understanding.