# From Melting Pots to Misrepresentations: Exploring Harms in Generative AI

arXiv: https://arxiv.org/abs/2403.10776 · HTML: https://mlsanity.com/paper/2403.10776/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 42 references that are in the map.
- First posted on arXiv 16 Mar 2024.

## Builds on (most cited first, 21 of 21)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale](https://mlsanity.com/paper/2211.03759/) (2022)
- [Stable Bias: Analyzing Societal Representations in Diffusion Models](https://mlsanity.com/paper/2303.11408/) (2023)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [Social Biases through the Text-to-Image Generation Lens](https://mlsanity.com/paper/2304.06034/) (2023)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [On Measures of Biases and Harms in NLP](https://mlsanity.com/paper/2108.03362/) (2021)
- [Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMs](https://mlsanity.com/paper/2310.07251/) (2023)
- [AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia](https://mlsanity.com/paper/2305.11844/) (2023)
- [ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languages](https://mlsanity.com/paper/2305.10510/) (2023)
- ['Person' == Light-skinned, Western Man, and Sexualization of Women of Color: Stereotypes in Stable Diffusion](https://mlsanity.com/paper/2310.19981/) (2023)
- [WEIRD FAccTs: How Western, Educated, Industrialized, Rich, and Democratic is FAccT?](https://mlsanity.com/paper/2305.06415/) (2023)
- [Automated Ableism: An Exploration of Explicit Disability Biases in Sentiment and Toxicity Analysis Models](https://mlsanity.com/paper/2307.09209/) (2023)
- [Unmasking Nationality Bias: A Study of Human Perception of Nationalities in AI-Generated Articles](https://mlsanity.com/paper/2308.04346/) (2023)
- [The "Colonial Impulse" of Natural Language Processing: An Audit of Bengali Sentiment Analysis Tools and Their Identity-based Biases](https://mlsanity.com/paper/2401.10535/) (2024)
- [Model Positionality and Computational Reflexivity: Promoting Reflexivity in Data Science](https://mlsanity.com/paper/2203.07031/) (2022)
- [The Sentiment Problem: A Critical Survey towards Deconstructing Sentiment Analysis](https://mlsanity.com/paper/2310.12318/) (2023)
- [Are We Asking the Right Questions?: Designing for Community Stakeholders' Interactions with AI in Policing](https://mlsanity.com/paper/2402.05348/) (2024)
- [Us vs. Them: A Dataset of Populist Attitudes, News Bias and Emotions](https://mlsanity.com/paper/2101.11956/) (2021)
- [Towards a Holistic Approach: Understanding Sociodemographic Biases in NLP Models using an Interdisciplinary Lens](https://mlsanity.com/paper/2308.13089/) (2023)

## Built on by (most cited first, 13 of 13)

- [LLMs are Biased Teachers: Evaluating LLM Bias in Personalized Education](https://mlsanity.com/paper/2410.14012/) (2024)
- [The Open Source Advantage in Large Language Models (LLMs)](https://mlsanity.com/paper/2412.12004/) (2024)
- [An Audit on the Perspectives and Challenges of Hallucinations in NLP](https://mlsanity.com/paper/2404.07461/) (2024)
- [Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responses](https://mlsanity.com/paper/2410.22349/) (2024)
- [A Tale of Two Identities: An Ethical Audit of Human and AI-Crafted Personas](https://mlsanity.com/paper/2505.07850/) (2025)
- [Detecting music deepfakes is easy but actually hard](https://mlsanity.com/paper/2405.04181/) (2024)
- [GenColor: Generative Color-Concept Association in Visual Design](https://mlsanity.com/paper/2503.03236/) (2025)
- [Multi-Group Proportional Representation in Retrieval](https://mlsanity.com/paper/2407.08571/) (2024)
- [Breaking Down Bias: On The Limits of Generalizable Pruning Strategies](https://mlsanity.com/paper/2502.07771/) (2025)
- [Race and Privacy in Broadcast Police Communications](https://mlsanity.com/paper/2407.01817/) (2024)
- [Interrogating Design Homogenization in Web Vibe Coding](https://mlsanity.com/paper/2603.13036/) (2026)
- [Investigating Social Bias in Narrative Image Generation](https://mlsanity.com/paper/2608.01780/) (2026)
- [Designing and Evaluating Chain-of-Hints for Scientific Question Answering](https://mlsanity.com/paper/2510.21087/) (2025)

## Abstract (arXiv)

With the widespread adoption of advanced generative models such as Gemini and GPT, there has been a notable increase in the incorporation of such models into sociotechnical systems, categorized under AI-as-a-Service (AIaaS). Despite their versatility across diverse sectors, concerns persist regarding discriminatory tendencies within these models, particularly favoring selected `majority' demographics across various sociodemographic dimensions. Despite widespread calls for diversification of media representations, marginalized racial and ethnic groups continue to face persistent distortion, stereotyping, and neglect within the AIaaS context. In this work, we provide a critical summary of the state of research in the context of social harms to lead the conversation to focus on their implications. We also present open-ended research questions, guided by our discussion, to help define future research pathways.