# Fair Sampling in Diffusion Models through Switching Mechanism

arXiv: https://arxiv.org/abs/2401.03140 · HTML: https://mlsanity.com/paper/2401.03140/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 6 Jan 2024.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [DiffEdit: Diffusion-based semantic image editing with mask guidance](https://mlsanity.com/paper/2210.11427/) (2022)
- [Diffusion Models already have a Semantic Latent Space](https://mlsanity.com/paper/2210.10960/) (2022)
- [Perception Prioritized Training of Diffusion Models](https://mlsanity.com/paper/2204.00227/) (2022)
- [FairGAN: Fairness-aware Generative Adversarial Networks](https://mlsanity.com/paper/1805.11202/) (2018)
- [Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness](https://mlsanity.com/paper/2302.10893/) (2023)
- [Fair Generative Modeling via Weak Supervision](https://mlsanity.com/paper/1910.12008/) (2019)
- [D2C: Diffusion-Denoising Models for Few-shot Conditional Generation](https://mlsanity.com/paper/2106.06819/) (2021)
- [Polarity Sampling: Quality and Diversity Control of Pre-Trained Generative Networks via Singular Values](https://mlsanity.com/paper/2203.01993/) (2022)
- [On the Fairness of Generative Adversarial Networks (GANs)](https://mlsanity.com/paper/2103.00950/) (2021)

## Built on by (most cited first, 12 of 12)

- [AIGC for Industrial Time Series: From Deep Generative Models to Large Generative Models](https://mlsanity.com/paper/2407.11480/) (2024)
- [How Fair is Your Diffusion Recommender Model?](https://mlsanity.com/paper/2409.04339/) (2024)
- [BlurGuard: A Simple Approach for Robustifying Image Protection Against AI-Powered Editing](https://mlsanity.com/paper/2511.00143/) (2025)
- [Multi-Group Proportional Representation for Text-to-Image Models](https://mlsanity.com/paper/2505.24023/) (2025)
- [Perceptual Fairness in Image Restoration](https://mlsanity.com/paper/2405.13805/) (2024)
- [Inference-Time Attribute Distribution Alignment for Unconditional Diffusion](https://mlsanity.com/paper/2605.07456/) (2026)
- [AITTI: Learning Adaptive Inclusive Token for Text-to-Image Generation](https://mlsanity.com/paper/2406.12805/) (2024)
- [Ctrl-Z Sampling: Scaling Diffusion Sampling with Controlled Random Zigzag Explorations](https://mlsanity.com/paper/2506.20294/) (2025)
- [Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization](https://mlsanity.com/paper/2607.00817/) (2026)
- [DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing](https://mlsanity.com/paper/2410.05694/) (2024)
- [FAROS: Fair Graph Generation via Attribute Switching Mechanisms](https://mlsanity.com/paper/2507.03728/) (2025)
- [Task-tailored Pre-processing: Fair Downstream Supervised Learning](https://mlsanity.com/paper/2601.11897/) (2026)

## Abstract (arXiv)

Diffusion models have shown their effectiveness in generation tasks by well-approximating the underlying probability distribution. However, diffusion models are known to suffer from an amplified inherent bias from the training data in terms of fairness. While the sampling process of diffusion models can be controlled by conditional guidance, previous works have attempted to find empirical guidance to achieve quantitative fairness. To address this limitation, we propose a fairness-aware sampling method called textitattribute switching mechanism for diffusion models. Without additional training, the proposed sampling can obfuscate sensitive attributes in generated data without relying on classifiers. We mathematically prove and experimentally demonstrate the effectiveness of the proposed method on two key aspects: (i) the generation of fair data and (ii) the preservation of the utility of the generated data.