# Rejected Dialects: Biases Against African American Language in Reward Models

arXiv: https://arxiv.org/abs/2502.12858 · HTML: https://mlsanity.com/paper/2502.12858/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 40 of its 77 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 30 of 40)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint](https://mlsanity.com/paper/2312.11456/) (2023)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://mlsanity.com/paper/2311.10702/) (2023)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Demographic Dialectal Variation in Social Media: A Case Study of African-American English](https://mlsanity.com/paper/1608.08868/) (2016)
- [Advancing LLM Reasoning Generalists with Preference Trees](https://mlsanity.com/paper/2404.02078/) (2024)
- [Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine](https://mlsanity.com/paper/2301.08745/) (2023)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs](https://mlsanity.com/paper/2406.10216/) (2024)
- [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022)
- [Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview](https://mlsanity.com/paper/1912.11078/) (2019)
- [Detoxifying Language Models Risks Marginalizing Minority Voices](https://mlsanity.com/paper/2104.06390/) (2021)
- [Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment](https://mlsanity.com/paper/2402.19085/) (2024)
- [Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback](https://mlsanity.com/paper/2406.09279/) (2024)
- [Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty](https://mlsanity.com/paper/2401.06730/) (2024)
- [Challenges in Automated Debiasing for Toxic Language Detection](https://mlsanity.com/paper/2102.00086/) (2021)
- [Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English](https://mlsanity.com/paper/1707.00061/) (2017)

## Built on by (most cited first, 20 of 20)

- [Detecting Prefix Bias in LLM-based Reward Models](https://mlsanity.com/paper/2505.13487/) (2025)
- [Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization](https://mlsanity.com/paper/2603.08091/) (2026)
- [Unsupervised Concept Vector Extraction for Bias Control in LLMs](https://mlsanity.com/paper/2502.19721/) (2025)
- [Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles](https://mlsanity.com/paper/2604.21152/) (2026)
- [Data Caricatures: On the Representation of African American Language in Pretraining Corpora](https://mlsanity.com/paper/2503.10789/) (2025)
- [EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferences](https://mlsanity.com/paper/2510.06370/) (2025)
- [Causal methods for LLM development and evaluation](https://mlsanity.com/paper/2605.25998/) (2026)
- [Structuring the Space of Sociotechnical Alignment](https://mlsanity.com/paper/2607.01250/) (2026)
- [Reward Model Perspectives: Whose Opinions Do Reward Models Reward?](https://mlsanity.com/paper/2510.06391/) (2025)
- [DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects](https://mlsanity.com/paper/2604.05318/) (2026)
- [DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation](https://mlsanity.com/paper/2607.07669/) (2026)
- [White-Box Sensitivity Auditing with Steering Vectors](https://mlsanity.com/paper/2601.16398/) (2026)
- [The Hrunting of AI: Where and How to Improve English Dialectal Fairness](https://mlsanity.com/paper/2603.15187/) (2026)
- [Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Once](https://mlsanity.com/paper/2604.01504/) (2026)
- [CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries](https://mlsanity.com/paper/2607.05405/) (2026)
- [LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steering](https://mlsanity.com/paper/2607.06845/) (2026)
- [Probing Latent Colombian Identity Inferences in Qwen2.5-7B with Natural Language Autoencoders](https://mlsanity.com/paper/2607.21774/) (2026)
- [Who Pays More for Safety? Measuring the Disparate Cost of Safety Alignment across Languages](https://mlsanity.com/paper/2608.22490/) (2026)
- [The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipeline](https://mlsanity.com/paper/2608.24952/) (2026)
- [Retrieval Sensitivity to Identity Signals in Queries](https://mlsanity.com/paper/2609.36534/) (2026)

## Abstract (arXiv)

Preference alignment via reward models helps build safe, helpful, and reliable large language models (LLMs). However, subjectivity in preference judgments and the lack of representative sampling in preference data collection can introduce new biases, hindering reward models' fairness and equity. In this work, we introduce a framework for evaluating dialect biases in reward models and conduct a case study on biases against African American Language (AAL) through several experiments comparing reward model preferences and behavior on paired White Mainstream English (WME) and both machine-translated and human-written AAL corpora. We show that reward models are less aligned with human preferences when processing AAL texts vs. WME ones (-4% accuracy on average), frequently disprefer AAL-aligned texts vs. WME-aligned ones, and steer conversations toward WME, even when prompted with AAL texts. Our findings provide a targeted analysis of anti-AAL biases at a relatively understudied stage in LLM development, highlighting representational harms and ethical questions about the desired behavior of LLMs concerning AAL.