mlsanityData to 7 Oct 2026

Safety and Fairness for Content Moderation in Generative Models

arXiv 2306.06135 ↗First posted 9 Jun 2023cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

With significant advances in generative AI, new technologies are rapidly being deployed with generative components. Generative models are typically trained on large datasets, resulting in model behaviors that can mimic the worst of the content in the training data. Responsible deployment of generative technologies requires content moderation strategies, such as safety input and output filters. Here, we provide a theoretical framework for conceptualizing responsible content moderation of text-to-image generative technologies, including a demonstration of how to empirically measure the constructs we enumerate. We define and distinguish the concepts of safety, fairness, and metric equity, and enumerate example harms that can come in each domain. We then provide a demonstration of how the defined harms can be quantified. We conclude with a summary of how the style of harms quantification we demonstrate enables data-driven content moderation decisions.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 30 of its 70 references that are in the map. First posted on arXiv 9 Jun 2023.

Builds on

30 of 70 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  4. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  5. 2017Algorithmic decision making and the cost of fairnesscited by 402
  6. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  7. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  8. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  9. 2022Red-Teaming the Stable Diffusion Safety Filtercited by 257
  10. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  11. 2020Towards Debiasing Sentence Representationscited by 170
  12. 2020Recipes for Safety in Open-domain Chatbotscited by 168
18 more
  1. 2017Data Decisions and Theoretical Implications when Adversarially Learning Fair Representationscited by 165
  2. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 150
  3. 2019Does Object Recognition Work for Everyone?cited by 149
  4. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  5. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  6. 2020Directions in Abusive Language Training Data: Garbage In, Garbage Outcited by 139
  7. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
  8. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  9. 2023Regulating ChatGPT and other Large Generative AI Modelscited by 102
  10. 2021On Measures of Biases and Harms in NLPcited by 81
  11. 2023The Gradient of Generative AI Release: Methods and Considerationscited by 63
  12. 2019Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Functioncited by 57
  13. 2022Measuring Representational Harms in Image Captioningcited by 36
  14. 2017Like trainer, like bot? Inheritance of bias in algorithmic content moderationcited by 34
  15. 2021Measuring Model Biases in the Absence of Ground Truthcited by 24
  16. 2022Cultural Incongruencies in Artificial Intelligencecited by 21
  17. 2022The Expertise Problem: Learning from Specialized Feedbackcited by 20
  18. 2023Toward General Design Principles for Generative AI Applicationscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 3; 2026: 2.

15 papers.

Sort:

By area:

  1. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 72
  2. 2024Content Moderation by LLM: From Accuracy to Legitimacycited by 30
  3. 2024ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generationcited by 27
  4. 2024Harm Amplification in Text-to-Image Modelscited by 14
  5. 2023Moderating Model Marketplaces: Platform Governance Puzzles for AI Intermediariescited by 9
  6. 2025Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generationcited by 9
  7. 2023Diffusion Models for Generative Artificial Intelligence: An Introduction for Applied Mathematicianscited by 5
  8. 2025Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluationscited by 4
  9. 2024Do not think about pink elephant!cited by 3
  10. 2024Data Pruning in Generative Diffusion Modelscited by 3
  11. 2024More is Less? A Simulation-Based Approach to Dynamic Interactions between Biases in Multimodal Modelscited by 1
  12. 2025What Is the Point of Equality in Machine Learning Fairness? Beyond Equality of Opportunitycited by 1
3 more
  1. 2026Bias at the End of the Scorecited by 1
  2. 2024AI red-teaming is a sociotechnical problem: on values, labor, and harmscited by 0
  3. 2026Not Safe for All: Auditing the Dialect Penalty in Text-to-Image Safety Pipelinescited by 0