# Analyzing Bias in Diffusion-based Face Generation Models

arXiv: https://arxiv.org/abs/2305.06402 · HTML: https://mlsanity.com/paper/2305.06402/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 10 May 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models](https://mlsanity.com/paper/2212.03860/) (2022)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [Perception Prioritized Training of Diffusion Models](https://mlsanity.com/paper/2204.00227/) (2022)
- [Stable Bias: Analyzing Societal Representations in Diffusion Models](https://mlsanity.com/paper/2303.11408/) (2023)
- [Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learning](https://mlsanity.com/paper/1911.10692/) (2019)
- [Social Biases through the Text-to-Image Generation Lens](https://mlsanity.com/paper/2304.06034/) (2023)
- [Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications](https://mlsanity.com/paper/2008.02793/) (2020)
- [Improving the Fairness of Deep Generative Models without Retraining](https://mlsanity.com/paper/2012.04842/) (2020)
- [Inclusive GAN: Improving Data and Minority Coverage in Generative Models](https://mlsanity.com/paper/2004.03355/) (2020)
- [Characterizing the Variability in Face Recognition Accuracy Relative to Race](https://mlsanity.com/paper/1904.07325/) (2019)
- [PASS: Protected Attribute Suppression System for Mitigating Bias in Face Recognition](https://mlsanity.com/paper/2108.03764/) (2021)
- [Synthetic Data for Face Recognition: Current State and Future Prospects](https://mlsanity.com/paper/2305.01021/) (2023)
- [How Does Gender Balance In Training Data Affect Face Recognition Accuracy?](https://mlsanity.com/paper/2002.02934/) (2020)
- [When do GANs replicate? On the choice of dataset size](https://mlsanity.com/paper/2202.11765/) (2022)
- [Gendered Differences in Face Recognition Accuracy Explained by Hairstyles, Makeup, and Facial Morphology](https://mlsanity.com/paper/2112.14656/) (2021)
- [Studying Bias in GANs through the Lens of Race](https://mlsanity.com/paper/2209.02836/) (2022)
- [Face Recognition Accuracy Across Demographics: Shining a Light Into the Problem](https://mlsanity.com/paper/2206.01881/) (2022)
- [RepFair-GAN: Mitigating Representation Bias in GANs Using Gradient Clipping](https://mlsanity.com/paper/2207.10653/) (2022)

## Built on by (most cited first, 30 of 46)

- [On the Design Fundamentals of Diffusion Models: A Survey](https://mlsanity.com/paper/2306.04542/) (2023)
- [Balancing Act: Distribution-Guided Debiasing in Diffusion Models](https://mlsanity.com/paper/2402.18206/) (2024)
- [Photoswap: Personalized Subject Swapping in Images](https://mlsanity.com/paper/2305.18286/) (2023)
- [The Journey, Not the Destination: How Data Guides Diffusion Models](https://mlsanity.com/paper/2312.06205/) (2023)
- [FairRAG: Fair Human Generation via Fair Retrieval Augmentation](https://mlsanity.com/paper/2403.19964/) (2024)
- [Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion](https://mlsanity.com/paper/2408.12692/) (2024)
- [Harm Amplification in Text-to-Image Models](https://mlsanity.com/paper/2402.01787/) (2024)
- [What do we learn from inverting CLIP models?](https://mlsanity.com/paper/2403.02580/) (2024)
- [Diffusion-driven GAN Inversion for Multi-Modal Face Image Generation](https://mlsanity.com/paper/2405.04356/) (2024)
- [Frame by Familiar Frame: Understanding Replication in Video Diffusion Models](https://mlsanity.com/paper/2403.19593/) (2024)
- [Replication in Visual Diffusion Models: A Survey and Outlook](https://mlsanity.com/paper/2408.00001/) (2024)
- [Beyond Real Faces: Synthetic Datasets Can Achieve Reliable Recognition Performance without Privacy Compromise](https://mlsanity.com/paper/2510.17372/) (2025)
- [Exploring Social Bias in Downstream Applications of Text-to-Image Foundation Models](https://mlsanity.com/paper/2312.10065/) (2023)
- [SDFD: Building a Versatile Synthetic Face Image Dataset with Diverse Attributes](https://mlsanity.com/paper/2404.17255/) (2024)
- [VersusDebias: Universal Zero-Shot Debiasing for Text-to-Image Models via SLM-Based Prompt Engineering and Generative Adversary](https://mlsanity.com/paper/2407.19524/) (2024)
- [Gaussian Harmony: Attaining Fairness in Diffusion-based Face Generation Models](https://mlsanity.com/paper/2312.14976/) (2023)
- [Debiasing Diffusion Model: Enhancing Fairness through Latent Representation Learning in Stable Diffusion Model](https://mlsanity.com/paper/2503.12536/) (2025)
- [RL4Med-DDPO: Reinforcement Learning for Controlled Guidance Towards Diverse Medical Image Generation using Vision-Language Foundation Models](https://mlsanity.com/paper/2503.15784/) (2025)
- [Limitations of Face Image Generation](https://mlsanity.com/paper/2309.07277/) (2023)
- [DiffSpectralNet : Unveiling the Potential of Diffusion Models for Hyperspectral Image Classification](https://mlsanity.com/paper/2312.12441/) (2023)
- [Diffusing DeBias: Synthetic Bias Amplification for Model Debiasing](https://mlsanity.com/paper/2502.09564/) (2025)
- [RespoDiff: Dual-Module Bottleneck Transformation for Responsible & Faithful T2I Generation](https://mlsanity.com/paper/2509.15257/) (2025)
- [Fully Unsupervised Self-debiasing of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2512.03749/) (2025)
- [Diverse Feature Learning by Self-distillation and Reset](https://mlsanity.com/paper/2403.19941/) (2024)
- [Tuning-Free Visual Customization via View Iterative Self-Attention Control](https://mlsanity.com/paper/2406.06258/) (2024)
- [The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation](https://mlsanity.com/paper/2407.12579/) (2024)
- [Fairer Analysis and Demographically Balanced Face Generation for Fairer Face Verification](https://mlsanity.com/paper/2412.03349/) (2024)
- [What's color got to do with it? Face recognition in grayscale](https://mlsanity.com/paper/2309.05180/) (2023)
- [Mitigating the Impact of Attribute Editing on Face Recognition](https://mlsanity.com/paper/2403.08092/) (2024)
- [Bias Begets Bias: The Impact of Biased Embeddings on Diffusion Models](https://mlsanity.com/paper/2409.09569/) (2024)

## Abstract (arXiv)

Diffusion models are becoming increasingly popular in synthetic data generation and image editing applications. However, these models can amplify existing biases and propagate them to downstream applications. Therefore, it is crucial to understand the sources of bias in their outputs. In this paper, we investigate the presence of bias in diffusion-based face generation models with respect to attributes such as gender, race, and age. Moreover, we examine how dataset size affects the attribute composition and perceptual quality of both diffusion and Generative Adversarial Network (GAN) based face generation models across various attribute classes. Our findings suggest that diffusion models tend to worsen distribution bias in the training data for various attributes, which is heavily influenced by the size of the dataset. Conversely, GAN models trained on balanced datasets with a larger number of samples show less bias across different attributes.