mlsanityData to 7 Oct 2026

Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness

arXiv 2302.10893 ↗First posted 7 Feb 2023cs.LG cs.AI cs.CVcited by 149

Not on the map. Why?

Cited by 61 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generative AI models have recently achieved astonishing results in quality and are consequently employed in a fast-growing number of applications. However, since they are highly data-driven, relying on billion-sized datasets randomly scraped from the internet, they also suffer from degenerated and biased human behavior, as we demonstrate. In fact, they may even reinforce such biases. To not only uncover but also combat these undesired effects, we present a novel strategy, called Fair Diffusion, to attenuate biases after the deployment of generative text-to-image models. Specifically, we demonstrate shifting a bias, based on human instructions, in any direction yielding arbitrarily new proportions for, e.g., identity groups. As our empirical evaluation demonstrates, this introduced control enables instructing generative image models on fairness, with no data filtering and additional training required.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 7 Feb 2023.

Builds on

37 of 51 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  11. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  12. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
25 more
  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  2. 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
  3. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  4. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  5. 2022Understanding Diffusion Models: A Unified Perspectivecited by 337
  6. 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
  7. 2015Censoring Representations with an Adversarycited by 191
  8. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  9. 2018FairGAN: Fairness-aware Generative Adversarial Networkscited by 164
  10. 2019Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchycited by 156
  11. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  12. 2017Fairness in Machine Learning: Lessons from Political Philosophycited by 148
  13. 2020Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biasescited by 107
  14. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  15. 201850 Years of Test (Un)fairness: Lessons for Machine Learningcited by 101
  16. 2022How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions?cited by 92
  17. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 84
  18. 2022Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluationcited by 59
  19. 2022Discover and Mitigate Unknown Biases with Debiasing Alternate Networkscited by 47
  20. 2022American == White in Multimodal Language-and-Image AIcited by 39
  21. 2022Markedness in Visual Semantic AIcited by 29
  22. 2022Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignoring the trollscited by 11
  23. 2022Revision Transformers: Instructing Language Models to Change their Valuescited by 8
  24. 2022A Typology for Exploring the Mitigation of Shortcut Behaviorcited by 7
  25. 2021The Myth of Complete AI-Fairnesscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 29; 2024: 57; 2025: 42; 2026: 21.

149 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  2. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 297
  3. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 266
  4. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  5. 2023LEDITS++: Limitless Image Editing using Text-to-Image Modelscited by 147
  6. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  7. 2023Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentationcited by 112
  8. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  9. 2023Finetuning Text-to-Image Diffusion Models for Fairnesscited by 90
  10. 2024Large-scale Reinforcement Learning for Diffusion Modelscited by 78
  11. 2023The Bias Amplification Paradox in Text-to-Image Generationcited by 69
  12. 2023Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generationcited by 69
38 more
  1. 2023ITI-GEN: Inclusive Text-to-Image Generationcited by 64
  2. 2023Toward effective protection against diffusion based mimicry through score distillationcited by 64
  3. 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
  4. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
  5. 2023TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Modelscited by 51
  6. 2024Balancing Act: Distribution-Guided Debiasing in Diffusion Modelscited by 50
  7. 2023Inspecting the Geographical Representativeness of Images from Text-to-Image Modelscited by 44
  8. 2024Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Imagescited by 41
  9. 2023Quality-Diversity through AI Feedbackcited by 40
  10. 2024FairRAG: Fair Human Generation via Fair Retrieval Augmentationcited by 34
  11. 2024Self-Improving Diffusion Models with Synthetic Datacited by 34
  12. 2023Fair Text-to-Image Diffusion via Fair Mappingcited by 31
  13. 2025FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Modelcited by 31
  14. 2023Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast Setscited by 27
  15. 2023Survey of Social Bias in Vision-Language Modelscited by 26
  16. 2024Training Unbiased Diffusion Models From Biased Datasetcited by 25
  17. 2024Would Deep Generative Models Amplify Bias in Future Models?cited by 24
  18. 2024Improving Geo-diversity of Generated Images with Contextualized Vendi Score Guidancecited by 24
  19. 2024Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusioncited by 22
  20. 2024Image Generation Diversity Issues and How to Tame Themcited by 22
  21. 2024Navigating Shortcuts, Spurious Correlations, and Confounders: From Origins via Detection to Mitigationcited by 22
  22. 2024FAIntbench: A Holistic and Precise Benchmark for Bias Evaluation in Text-to-Image Modelscited by 21
  23. 2023Mitigating stereotypical biases in text to image generative systemscited by 19
  24. 2024Fairness and Bias in Multimodal AI: A Surveycited by 19
  25. 2024SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIPcited by 19
  26. 2024Constrained Diffusion Models via Dual Trainingcited by 19
  27. 2024Minority-Focused Text-to-Image Generation via Prompt Optimizationcited by 18
  28. 2024BIGbench: A Unified Benchmark for Evaluating Multi-dimensional Social Biases in Text-to-Image Modelscited by 17
  29. 2024BendVLM: Test-Time Debiasing of Vision-Language Embeddingscited by 17
  30. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15
  31. 2024Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help Youcited by 15
  32. 2025Synthetic Video Enhances Physical Fidelity in Video Synthesiscited by 15
  33. 2023MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generationcited by 14
  34. 2024AI-generated faces influence gender stereotypes and racial homogenizationcited by 13
  35. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
  36. 2024Fair Sampling in Diffusion Models through Switching Mechanismcited by 12
  37. 2024Balanced Mixed-Type Tabular Data Synthesis with Diffusion Modelscited by 12
  38. 2024Backdooring Bias (B2) into Stable Diffusion Modelscited by 12

50 most cited of 149 All 149 (JSON)