# Fair Text-to-Image Diffusion via Fair Mapping

arXiv: https://arxiv.org/abs/2311.17695 · HTML: https://mlsanity.com/paper/2311.17695/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 51 of its 66 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 51)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Pix2Video: Video Editing using Image Diffusion](https://mlsanity.com/paper/2303.12688/) (2023)
- [Text-to-image Diffusion Models in Generative AI: A Survey](https://mlsanity.com/paper/2303.07909/) (2023)
- [SpaText: Spatio-Textual Representation for Controllable Image Generation](https://mlsanity.com/paper/2211.14305/) (2022)

## Built on by (most cited first, 30 of 31)

- [Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation](https://mlsanity.com/paper/2404.01030/) (2024)
- [SATO: Stable Text-to-Motion Framework](https://mlsanity.com/paper/2405.01461/) (2024)
- [Text Guided Image Editing with Automatic Concept Locating and Forgetting](https://mlsanity.com/paper/2405.19708/) (2024)
- [Tackling Noisy Clients in Federated Learning with End-to-end Label Correction](https://mlsanity.com/paper/2408.04301/) (2024)
- [Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study](https://mlsanity.com/paper/2412.18260/) (2024)
- [FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models](https://mlsanity.com/paper/2510.21363/) (2025)
- [Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Survey](https://mlsanity.com/paper/2409.18214/) (2024)
- [FNBench: Benchmarking Robust Federated Learning against Noisy Labels](https://mlsanity.com/paper/2505.06684/) (2025)
- [FairSkin: Fair Diffusion for Skin Disease Image Generation](https://mlsanity.com/paper/2410.22551/) (2024)
- [TransDiffuser: Diverse Trajectory Generation with Decorrelated Multi-modal Representation for End-to-end Autonomous Driving](https://mlsanity.com/paper/2505.09315/) (2025)
- [TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Models](https://mlsanity.com/paper/2601.08011/) (2026)
- [Multi-Group Proportional Representation for Text-to-Image Models](https://mlsanity.com/paper/2505.24023/) (2025)
- [Beyond Overcorrection: Evaluating Diversity in T2I Models with DivBench](https://mlsanity.com/paper/2507.03015/) (2025)
- [FAME: Fairness-aware Attention-modulated Video Editing](https://mlsanity.com/paper/2510.22960/) (2025)
- [Inference-Time Attribute Distribution Alignment for Unconditional Diffusion](https://mlsanity.com/paper/2605.07456/) (2026)
- [Stay Fair! Ensuring Group Fairness in Diffusion Models Across Guidance Scales](https://mlsanity.com/paper/2605.28036/) (2026)
- [Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generation](https://mlsanity.com/paper/2607.14962/) (2026)
- [AITTI: Learning Adaptive Inclusive Token for Text-to-Image Generation](https://mlsanity.com/paper/2406.12805/) (2024)
- [What Is the Point of Equality in Machine Learning Fairness? Beyond Equality of Opportunity](https://mlsanity.com/paper/2506.16782/) (2025)
- [Prompting Away Stereotypes? Evaluating Bias in Text-to-Image Models for Occupations](https://mlsanity.com/paper/2509.00849/) (2025)
- [ZoFia: Zero-Shot Fake News Detection with Entity-Guided Retrieval and Multi-LLM Interaction](https://mlsanity.com/paper/2511.01188/) (2025)
- [HoloFair: Unified T2I Fairness Evaluation and Fair-GRPO Debiasing](https://mlsanity.com/paper/2605.24687/) (2026)
- [Training-Free Debiasing of Diffusion Models via CLIP-Guided Denoising Optimization](https://mlsanity.com/paper/2607.00817/) (2026)
- [A Framework for Benchmarking Fairness-Utility Trade-offs in Text-to-Image Models via Pareto Frontiers](https://mlsanity.com/paper/2508.16752/) (2025)
- [FairJudge: Abstention-Aware Multimodal Judges for Fairness and Alignment Evaluation in Text-to-Image Models](https://mlsanity.com/paper/2510.22827/) (2025)
- [Operationalizing Fairness in Text-to-Image Models: A Survey of Bias, Fairness Audits and Mitigation Strategies](https://mlsanity.com/paper/2604.16516/) (2026)
- [Orthogonal Negative Guidance in Attention Feature Space for Text-to-Image Generation](https://mlsanity.com/paper/2605.29390/) (2026)
- [KG-FairDiff: Knowledge Graph-Guided Prompt Refinement for Demographically Fair Text-to-Image Generation](https://mlsanity.com/paper/2606.01282/) (2026)
- [Intermediate Text Representation Guided Text-to-Image Generation for Enhancing One-and-Only Alignment](https://mlsanity.com/paper/2606.30262/) (2026)
- [FairFlow: Demystifying and Mitigating Stereotype Bias in Text-to-Image Diffusion Transformers](https://mlsanity.com/paper/2607.03180/) (2026)

## Abstract (arXiv)

In this paper, we address the limitations of existing text-to-image diffusion models in generating demographically fair results when given human-related descriptions. These models often struggle to disentangle the target language context from sociocultural biases, resulting in biased image generation. To overcome this challenge, we propose Fair Mapping, a flexible, model-agnostic, and lightweight approach that modifies a pre-trained text-to-image diffusion model by controlling the prompt to achieve fair image generation. One key advantage of our approach is its high efficiency. It only requires updating an additional linear network with few parameters at a low computational cost. By developing a linear network that maps conditioning embeddings into a debiased space, we enable the generation of relatively balanced demographic results based on the specified text condition. With comprehensive experiments on face image generation, we show that our method significantly improves image generation fairness with almost the same image quality compared to conventional diffusion models when prompted with descriptions related to humans. By effectively addressing the issue of implicit language bias, our method produces more fair and diverse image outputs.