# Hey GPT, Can You be More Racist? Analysis from Crowdsourced Attempts to Elicit Biased Content from Generative AI

arXiv: https://arxiv.org/abs/2410.15467 · HTML: https://mlsanity.com/paper/2410.15467/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 41 references that are in the map.
- First posted on arXiv 20 Oct 2024.

## Builds on (most cited first, 18 of 18)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- ["I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://mlsanity.com/paper/2205.09209/) (2022)
- [Aequitas: A Bias and Fairness Audit Toolkit](https://mlsanity.com/paper/1811.05577/) (2018)
- [The Frontiers of Fairness in Machine Learning](https://mlsanity.com/paper/1810.08810/) (2018)
- [Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era](https://mlsanity.com/paper/2404.11457/) (2024)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [On Measures of Biases and Harms in NLP](https://mlsanity.com/paper/2108.03362/) (2021)
- [Disclosure and Mitigation of Gender Bias in LLMs](https://mlsanity.com/paper/2402.11190/) (2024)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMs](https://mlsanity.com/paper/2404.14461/) (2024)
- [Do Generative AI Models Output Harm while Representing Non-Western Cultures: Evidence from A Community-Centered Approach](https://mlsanity.com/paper/2407.14779/) (2024)
- [Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition](https://mlsanity.com/paper/2406.07954/) (2024)
- [Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI](https://mlsanity.com/paper/2409.15398/) (2024)
- [Towards a Holistic Approach: Understanding Sociodemographic Biases in NLP Models using an Interdisciplinary Lens](https://mlsanity.com/paper/2308.13089/) (2023)
- [Are Large Language Models Really Bias-Free? Jailbreak Prompts for Assessing Adversarial Robustness to Bias Elicitation](https://mlsanity.com/paper/2407.08441/) (2024)

## Built on by (most cited first, 6 of 6)

- [Understanding and Mitigating Bias Inheritance in LLM-based Data Augmentation on Downstream Tasks](https://mlsanity.com/paper/2502.04419/) (2025)
- [Attention Pruning: Automated Fairness Repair of Language Models via Surrogate Simulated Annealing](https://mlsanity.com/paper/2503.15815/) (2025)
- [On the Performance of LLMs for Real Estate Appraisal](https://mlsanity.com/paper/2506.11812/) (2025)
- [To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems](https://mlsanity.com/paper/2604.28053/) (2026)
- [Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet](https://mlsanity.com/paper/2502.05291/) (2025)
- [Beyond Benchmarks: Exposing the Hidden Crisis in Bangla Hate Speech Detection](https://mlsanity.com/paper/2607.11597/) (2026)

## Abstract (arXiv)

The widespread adoption of large language models (LLMs) and generative AI (GenAI) tools across diverse applications has amplified the importance of addressing societal biases inherent within these technologies. While the NLP community has extensively studied LLM bias, research investigating how non-expert users perceive and interact with biases from these systems remains limited. As these technologies become increasingly prevalent, understanding this question is crucial to inform model developers in their efforts to mitigate bias. To address this gap, this work presents the findings from a university-level competition, which challenged participants to design prompts for eliciting biased outputs from GenAI tools. We quantitatively and qualitatively analyze the competition submissions and identify a diverse set of biases in GenAI and strategies employed by participants to induce bias in GenAI. Our finding provides unique insights into how non-expert users perceive and interact with biases from GenAI tools.