# Do Generative AI Models Output Harm while Representing Non-Western Cultures: Evidence from A Community-Centered Approach

arXiv: https://arxiv.org/abs/2407.14779 · HTML: https://mlsanity.com/paper/2407.14779/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 85 references that are in the map.
- First posted on arXiv 20 Jul 2024.

## Builds on (most cited first, 18 of 18)

- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale](https://mlsanity.com/paper/2211.03759/) (2022)
- [On Measuring and Mitigating Biased Inferences of Word Embeddings](https://mlsanity.com/paper/1908.09369/) (2019)
- [Re-imagining Algorithmic Fairness in India and Beyond](https://mlsanity.com/paper/2101.09995/) (2021)
- [Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence](https://mlsanity.com/paper/2007.04068/) (2020)
- [On Measures of Biases and Harms in NLP](https://mlsanity.com/paper/2108.03362/) (2021)
- [A Survey of Race, Racism, and Anti-Racism in NLP](https://mlsanity.com/paper/2106.11410/) (2021)
- [AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia](https://mlsanity.com/paper/2305.11844/) (2023)
- [WEIRD FAccTs: How Western, Educated, Industrialized, Rich, and Democratic is FAccT?](https://mlsanity.com/paper/2305.06415/) (2023)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [Automated Ableism: An Exploration of Explicit Disability Biases in Sentiment and Toxicity Analysis Models](https://mlsanity.com/paper/2307.09209/) (2023)
- [An Audit on the Perspectives and Challenges of Hallucinations in NLP](https://mlsanity.com/paper/2404.07461/) (2024)
- [Unmasking Nationality Bias: A Study of Human Perception of Nationalities in AI-Generated Articles](https://mlsanity.com/paper/2308.04346/) (2023)
- [The "Colonial Impulse" of Natural Language Processing: An Audit of Bengali Sentiment Analysis Tools and Their Identity-based Biases](https://mlsanity.com/paper/2401.10535/) (2024)
- [Out of Context: Investigating the Bias and Fairness Concerns of "Artificial Intelligence as a Service"](https://mlsanity.com/paper/2302.01448/) (2023)
- [CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias](https://mlsanity.com/paper/2308.12539/) (2023)
- [The Sentiment Problem: A Critical Survey towards Deconstructing Sentiment Analysis](https://mlsanity.com/paper/2310.12318/) (2023)

## Built on by (most cited first, 26 of 26)

- [Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)](https://mlsanity.com/paper/2505.21091/) (2025)
- [The Case for "Thick Evaluations" of Cultural Representation in AI](https://mlsanity.com/paper/2503.19075/) (2025)
- [Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responses](https://mlsanity.com/paper/2410.22349/) (2024)
- [Can LLMs Express Personality Across Cultures? Introducing CulturalPersonas for Evaluating Trait Alignment](https://mlsanity.com/paper/2506.05670/) (2025)
- [A Tale of Two Identities: An Ethical Audit of Human and AI-Crafted Personas](https://mlsanity.com/paper/2505.07850/) (2025)
- [Hey GPT, Can You be More Racist? Analysis from Crowdsourced Attempts to Elicit Biased Content from Generative AI](https://mlsanity.com/paper/2410.15467/) (2024)
- [Invisible Filters: Cultural Bias in Hiring Evaluations Using Large Language Models](https://mlsanity.com/paper/2508.16673/) (2025)
- [KAHANI: Culturally-Nuanced Visual Storytelling Tool for Non-Western Cultures](https://mlsanity.com/paper/2410.19419/) (2024)
- [Through the Prism of Culture: Evaluating LLMs' Understanding of Indian Subcultures and Traditions](https://mlsanity.com/paper/2501.16748/) (2025)
- [RusCode: Russian Cultural Code Benchmark for Text-to-Image Generation](https://mlsanity.com/paper/2502.07455/) (2025)
- [What Is the Point of Equality in Machine Learning Fairness? Beyond Equality of Opportunity](https://mlsanity.com/paper/2506.16782/) (2025)
- [Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global South](https://mlsanity.com/paper/2605.19190/) (2026)
- [Prompt Revision as a Source of Cultural Bias in Text-to-Image Systems](https://mlsanity.com/paper/2609.11532/) (2026)
- [Sometimes the Model doth Preach: Quantifying Religious Bias in Open LLMs through Demographic Analysis in Asian Nations](https://mlsanity.com/paper/2503.07510/) (2025)
- [Text-to-Image Models and Their Representation of People from Different Nationalities Engaging in Activities](https://mlsanity.com/paper/2504.06313/) (2025)
- [NLP Meets the World: Toward Improving Conversations With the Public About Natural Language Processing Research](https://mlsanity.com/paper/2507.10559/) (2025)
- [Designing and Evaluating Chain-of-Hints for Scientific Question Answering](https://mlsanity.com/paper/2510.21087/) (2025)
- [Responsible AI for General-Purpose Systems: Overview, Challenges, and A Path Forward](https://mlsanity.com/paper/2601.13122/) (2026)
- [MAPLE: Metadata Conditioned LLM Pretraining for Locale-Aware Question Answering](https://mlsanity.com/paper/2601.15236/) (2026)
- [Operationalizing Fairness in Text-to-Image Models: A Survey of Bias, Fairness Audits and Mitigation Strategies](https://mlsanity.com/paper/2604.16516/) (2026)
- [Mod-Guide: An LLM-based Content Moderation Feedback System to Address Insensitive Speech toward Indigenous Ethnic and Religious Minority Communities](https://mlsanity.com/paper/2606.13397/) (2026)
- [The Market in the Model: Latent Diffusion as Neural Economy](https://mlsanity.com/paper/2606.19151/) (2026)
- [The Crowded Embedding Space: A Mean-Field Mechanism for Emergent Marginalization in Retrieval-Augmented Agents](https://mlsanity.com/paper/2606.28343/) (2026)
- [Harm is not Universal: Community-Specific Toxicity Detection is Urgently Needed](https://mlsanity.com/paper/2607.24898/) (2026)
- [Identifying Harm in Personalized, Generative AI Systems Requires User-Centered Auditing at the Interaction Level](https://mlsanity.com/paper/2608.14692/) (2026)
- [Knowing Is Not Telling: Identity Ossification in LLM Representations of Trans-border Communities](https://mlsanity.com/paper/2610.12610/) (2026)

## Abstract (arXiv)

Our research investigates the impact of Generative Artificial Intelligence (GAI) models, specifically text-to-image generators (T2Is), on the representation of non-Western cultures, with a focus on Indian contexts. Despite the transformative potential of T2Is in content creation, concerns have arisen regarding biases that may lead to misrepresentations and marginalizations. Through a community-centered approach and grounded theory analysis of 5 focus groups from diverse Indian subcultures, we explore how T2I outputs to English prompts depict Indian culture and its subcultures, uncovering novel representational harms such as exoticism and cultural misappropriation. These findings highlight the urgent need for inclusive and culturally sensitive T2I systems. We propose design guidelines informed by a sociotechnical perspective, aiming to address these issues and contribute to the development of more equitable and representative GAI technologies globally. Our work also underscores the necessity of adopting a community-centered approach to comprehend the sociotechnical dynamics of these models, complementing existing work in this space while identifying and addressing the potential negative repercussions and harms that may arise when these models are deployed on a global scale.