# Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models

arXiv: https://arxiv.org/abs/2304.03738 · HTML: https://mlsanity.com/paper/2304.03738/

## Key facts

- Cited by 148 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 163 references that are in the map.
- First posted on arXiv 7 Apr 2023.

## Builds on (most cited first, 30 of 74)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)

## Built on by (most cited first, 30 of 148)

- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use](https://mlsanity.com/paper/2310.03128/) (2023)
- [Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs](https://mlsanity.com/paper/2308.13387/) (2023)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection](https://mlsanity.com/paper/2307.16888/) (2023)
- [Cultural Bias and Cultural Alignment of Large Language Models](https://mlsanity.com/paper/2311.14096/) (2023)
- [Factuality Challenges in the Era of Large Language Models](https://mlsanity.com/paper/2310.05189/) (2023)
- [DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models](https://mlsanity.com/paper/2310.00902/) (2023)
- [Emergent social conventions and collective bias in LLM populations](https://mlsanity.com/paper/2410.08948/) (2024)
- [Generative AI](https://mlsanity.com/paper/2309.07930/) (2023)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [Learning Linear Causal Representations from Interventions under General Nonlinear Mixing](https://mlsanity.com/paper/2306.02235/) (2023)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [LLM Generated Persona is a Promise with a Catch](https://mlsanity.com/paper/2503.16527/) (2025)
- [Bias in Large Language Models: Origin, Evaluation, and Mitigation](https://mlsanity.com/paper/2411.10915/) (2024)
- [Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems](https://mlsanity.com/paper/2401.05778/) (2024)
- [Understanding LLMs: A Comprehensive Overview from Training to Inference](https://mlsanity.com/paper/2401.02038/) (2024)
- [Fairness in Large Language Models: A Taxonomic Survey](https://mlsanity.com/paper/2404.01349/) (2024)
- [Language Repository for Long Video Understanding](https://mlsanity.com/paper/2403.14622/) (2024)
- [FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models](https://mlsanity.com/paper/2309.05274/) (2023)
- [A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine](https://mlsanity.com/paper/2405.08603/) (2024)
- [IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models](https://mlsanity.com/paper/2406.03368/) (2024)

## Abstract (arXiv)

As the capabilities of generative language models continue to advance, the implications of biases ingrained within these models have garnered increasing attention from researchers, practitioners, and the broader public. This article investigates the challenges and risks associated with biases in large-scale language models like ChatGPT. We discuss the origins of biases, stemming from, among others, the nature of training data, model specifications, algorithmic constraints, product design, and policy decisions. We explore the ethical concerns arising from the unintended consequences of biased model outputs. We further analyze the potential opportunities to mitigate biases, the inevitability of some biases, and the implications of deploying these models in various applications, such as virtual assistants, content generation, and chatbots. Finally, we review the current approaches to identify, quantify, and mitigate biases in language models, emphasizing the need for a multi-disciplinary, collaborative effort to develop more equitable, transparent, and responsible AI systems. This article aims to stimulate a thoughtful dialogue within the artificial intelligence community, encouraging researchers and developers to reflect on the role of biases in generative language models and the ongoing pursuit of ethical AI.