# Conditioning Diffusion Models via Attributes and Semantic Masks for Face Generation

arXiv: https://arxiv.org/abs/2306.00914 · HTML: https://mlsanity.com/paper/2306.00914/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 41 of its 45 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [Tackling the Generative Learning Trilemma with Denoising Diffusion GANs](https://mlsanity.com/paper/2112.07804/) (2021)
- [Diffusion Autoencoders: Toward a Meaningful and Decodable Representation](https://mlsanity.com/paper/2111.15640/) (2021)
- [Perception Prioritized Training of Diffusion Models](https://mlsanity.com/paper/2204.00227/) (2022)
- [Toward Multimodal Image-to-Image Translation](https://mlsanity.com/paper/1711.11586/) (2017)
- [SEAN: Image Synthesis with Semantic Region-Adaptive Normalization](https://mlsanity.com/paper/1911.12861/) (2019)
- [Controllable Text-to-Image Generation](https://mlsanity.com/paper/1909.07083/) (2019)
- [StyleSwin: Transformer-based GAN for High-resolution Image Generation](https://mlsanity.com/paper/2112.10762/) (2021)
- [Semantic Image Synthesis via Diffusion Models](https://mlsanity.com/paper/2207.00050/) (2022)
- [Wavelet Diffusion Models are fast and scalable Image Generators](https://mlsanity.com/paper/2211.16152/) (2022)
- [Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis](https://mlsanity.com/paper/1910.06809/) (2019)
- [Diversity-Sensitive Conditional Generative Adversarial Networks](https://mlsanity.com/paper/1901.09024/) (2019)
- [Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance](https://mlsanity.com/paper/2210.05559/) (2022)

## Built on by (most cited first, 9 of 9)

- [Controllable Generation with Text-to-Image Diffusion Models: A Survey](https://mlsanity.com/paper/2403.04279/) (2024)
- [Multi-conditioned Graph Diffusion for Neural Architecture Search](https://mlsanity.com/paper/2403.06020/) (2024)
- [DynamicID: Zero-Shot Multi-ID Image Personalization with Flexible Facial Editability](https://mlsanity.com/paper/2503.06505/) (2025)
- [Prompting for products: Investigating design space exploration strategies for text-to-image generative models](https://mlsanity.com/paper/2408.03946/) (2024)
- [Enriching Information and Preserving Semantic Consistency in Expanding Curvilinear Object Segmentation Datasets](https://mlsanity.com/paper/2407.08209/) (2024)
- [Class-specific diffusion models improve military object detection in a low-data domain](https://mlsanity.com/paper/2604.18076/) (2026)
- [Minimum Data, Maximum Impact: 20 annotated samples for explainable lung nodule classification](https://mlsanity.com/paper/2508.00639/) (2025)
- [Improving Fine-Grained Control via Aggregation of Multiple Diffusion Models](https://mlsanity.com/paper/2410.01262/) (2024)
- [Enhancing Diffusion Face Generation with Contrastive Embeddings and SegFormer Guidance](https://mlsanity.com/paper/2508.09847/) (2025)

## Abstract (arXiv)

Deep generative models have shown impressive results in generating realistic images of faces. GANs managed to generate high-quality, high-fidelity images when conditioned on semantic masks, but they still lack the ability to diversify their output. Diffusion models partially solve this problem and are able to generate diverse samples given the same condition. In this paper, we propose a multi-conditioning approach for diffusion models via cross-attention exploiting both attributes and semantic masks to generate high-quality and controllable face images. We also studied the impact of applying perceptual-focused loss weighting into the latent space instead of the pixel space. Our method extends the previous approaches by introducing conditioning on more than one set of features, guaranteeing a more fine-grained control over the generated face images. We evaluate our approach on the CelebA-HQ dataset, and we show that it can generate realistic and diverse samples while allowing for fine-grained control over multiple attributes and semantic regions. Additionally, we perform an ablation study to evaluate the impact of different conditioning strategies on the quality and diversity of the generated images.