# Dialect prejudice predicts AI decisions about people's character, employability, and criminality

arXiv: https://arxiv.org/abs/2403.00742 · HTML: https://mlsanity.com/paper/2403.00742/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 109 references that are in the map.
- First posted on arXiv 1 Mar 2024.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Holistic Evaluation of Language Models](https://mlsanity.com/paper/2211.09110/) (2022)
- [Probing Image-Language Transformers for Verb Understanding](https://mlsanity.com/paper/2106.09141/) (2021)
- [Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English](https://mlsanity.com/paper/1707.00061/) (2017)
- [Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence](https://mlsanity.com/paper/2306.07075/) (2023)
- [Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias](https://mlsanity.com/paper/2212.10678/) (2022)
- [Discovering and Categorising Language Biases in Reddit](https://mlsanity.com/paper/2008.02754/) (2020)

## Built on by (most cited first, 30 of 37)

- [Lessons from the Trenches on Reproducible Evaluation of Language Models](https://mlsanity.com/paper/2405.14782/) (2024)
- [AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuances](https://mlsanity.com/paper/2409.11360/) (2024)
- [Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination](https://mlsanity.com/paper/2406.08818/) (2024)
- [Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas](https://mlsanity.com/paper/2408.06929/) (2024)
- [MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs](https://mlsanity.com/paper/2406.07243/) (2024)
- [Stereotype or Personalization? User Identity Biases Chatbot Recommendations](https://mlsanity.com/paper/2410.05613/) (2024)
- [Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression](https://mlsanity.com/paper/2407.04965/) (2024)
- ["They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversations](https://mlsanity.com/paper/2405.05378/) (2024)
- [Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks](https://mlsanity.com/paper/2410.11005/) (2024)
- [Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)](https://mlsanity.com/paper/2407.12858/) (2024)
- [Rejected Dialects: Biases Against African American Language in Reward Models](https://mlsanity.com/paper/2502.12858/) (2025)
- [ChatGPT Doesn't Trust Chargers Fans: Guardrail Sensitivity in Context](https://mlsanity.com/paper/2407.06866/) (2024)
- [Implicit Bias in LLMs: A Survey](https://mlsanity.com/paper/2503.02776/) (2025)
- [Aligning to What? Limits to RLHF Based Alignment](https://mlsanity.com/paper/2503.09025/) (2025)
- [Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese](https://mlsanity.com/paper/2505.22645/) (2025)
- [Do language models practice what they preach? Examining language ideologies about gendered language reform encoded in LLMs](https://mlsanity.com/paper/2409.13852/) (2024)
- [EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Models](https://mlsanity.com/paper/2504.07100/) (2025)
- [Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variations](https://mlsanity.com/paper/2602.01030/) (2026)
- [The Life Cycle of Large Language Models: A Review of Biases in Education](https://mlsanity.com/paper/2407.11203/) (2024)
- [Examining Multimodal Gender and Content Bias in ChatGPT-4o](https://mlsanity.com/paper/2411.19140/) (2024)
- [MEDEQUALQA: Evaluating Biases in LLMs with Counterfactual Reasoning](https://mlsanity.com/paper/2510.12818/) (2025)
- [A Taxonomy of Challenges to Curating Fair Datasets](https://mlsanity.com/paper/2406.06407/) (2024)
- [What should I wear to a party in a Greek taverna? Evaluation for Conversational Agents in the Fashion Domain](https://mlsanity.com/paper/2408.08907/) (2024)
- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)
- [The MERIT Dataset: Modelling and Efficiently Rendering Interpretable Transcripts](https://mlsanity.com/paper/2409.00447/) (2024)
- [An Empirically-grounded tool for Automatic Prompt Linting and Repair: A Case Study on Bias, Vulnerability, and Optimization in Developer Prompts](https://mlsanity.com/paper/2501.12521/) (2025)
- [My LLM might Mimic AAE -- But When Should it?](https://mlsanity.com/paper/2502.04564/) (2025)
- [Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English](https://mlsanity.com/paper/2503.12858/) (2025)
- [Mitigating LLM Hallucinations with Knowledge Graphs: A Case Study](https://mlsanity.com/paper/2504.12422/) (2025)
- [I Think, Therefore I Am Under-Qualified? A Benchmark for Evaluating Linguistic Shibboleth Detection in LLM Hiring Evaluations](https://mlsanity.com/paper/2508.04939/) (2025)

## Abstract (arXiv)

Hundreds of millions of people now interact with language models, with uses ranging from serving as a writing aid to informing hiring decisions. Yet these language models are known to perpetuate systematic racial prejudices, making their judgments biased in problematic ways about groups like African Americans. While prior research has focused on overt racism in language models, social scientists have argued that racism with a more subtle character has developed over time. It is unknown whether this covert racism manifests in language models. Here, we demonstrate that language models embody covert racism in the form of dialect prejudice: we extend research showing that Americans hold raciolinguistic stereotypes about speakers of African American English and find that language models have the same prejudice, exhibiting covert stereotypes that are more negative than any human stereotypes about African Americans ever experimentally recorded, although closest to the ones from before the civil rights movement. By contrast, the language models' overt stereotypes about African Americans are much more positive. We demonstrate that dialect prejudice has the potential for harmful consequences by asking language models to make hypothetical decisions about people, based only on how they speak. Language models are more likely to suggest that speakers of African American English be assigned less prestigious jobs, be convicted of crimes, and be sentenced to death. Finally, we show that existing methods for alleviating racial bias in language models such as human feedback training do not mitigate the dialect prejudice, but can exacerbate the discrepancy between covert and overt stereotypes, by teaching language models to superficially conceal the racism that they maintain on a deeper level. Our findings have far-reaching implications for the fair and safe employment of language technology.