mlsanityData to 14 Oct 2026

Rejected Dialects: Biases Against African American Language in Reward Models

arXiv 2502.12858 ↗First posted 18 Feb 2025cs.CL cs.AI cs.CYcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Preference alignment via reward models helps build safe, helpful, and reliable large language models (LLMs). However, subjectivity in preference judgments and the lack of representative sampling in preference data collection can introduce new biases, hindering reward models' fairness and equity. In this work, we introduce a framework for evaluating dialect biases in reward models and conduct a case study on biases against African American Language (AAL) through several experiments comparing reward model preferences and behavior on paired White Mainstream English (WME) and both machine-translated and human-written AAL corpora. We show that reward models are less aligned with human preferences when processing AAL texts vs. WME ones (-4% accuracy on average), frequently disprefer AAL-aligned texts vs. WME-aligned ones, and steer conversations toward WME, even when prompted with AAL texts. Our findings provide a targeted analysis of anti-AAL biases at a relatively understudied stage in LLM development, highlighting representational harms and ethical questions about the desired behavior of LLMs concerning AAL.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 77 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

40 of 77 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Models12 Oct
  2. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values11 Oct

Other references · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  7. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  10. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  11. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  12. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
26 more
  1. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  2. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  3. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  4. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 209
  5. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 196
  6. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  7. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
  8. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  9. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  10. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 138
  11. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  12. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 107
  13. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 107
  14. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  15. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  16. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 96
  17. 2017Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American Englishcited by 71
  18. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  19. 2024Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discriminationcited by 56
  20. 2020Investigating African-American Vernacular English in Transformer-Based Text Generationcited by 53
  21. 2022VALUE: Understanding Dialect Disparity in NLUcited by 47
  22. 2024sDPO: Don't Use Your Data All at Oncecited by 46
  23. 2024Dialect prejudice predicts AI decisions about people's character, employability, and criminalitycited by 37
  24. 2023Evaluation of African American Language Bias in Natural Language Generationcited by 36
  25. 2023Quantifying the Dialect Gap and its Correlates Across Languagescited by 33
  26. 2024"For Us By Us": Intentionally Designing Technology for Lived Black Experiencescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 15.

20 papers.

Sort:

By area:

  1. 2025Detecting Prefix Bias in LLM-based Reward Modelscited by 11
  2. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 9
  3. 2025Unsupervised Concept Vector Extraction for Bias Control in LLMscited by 7
  4. 2026Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profilescited by 4
  5. 2025Data Caricatures: On the Representation of African American Language in Pretraining Corporacited by 3
  6. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferencescited by 3
  7. 2026Causal methods for LLM development and evaluationcited by 2
  8. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  9. 2025Reward Model Perspectives: Whose Opinions Do Reward Models Reward?cited by 1
  10. 2026DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialectscited by 1
  11. 2026DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptationcited by 1
  12. 2026White-Box Sensitivity Auditing with Steering Vectorscited by 0
8 more
  1. 2026The Hrunting of AI: Where and How to Improve English Dialectal Fairnesscited by 0
  2. 2026Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Oncecited by 0
  3. 2026CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queriescited by 0
  4. 2026LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steeringcited by 0
  5. 2026Probing Latent Colombian Identity Inferences in Qwen2.5-7B with Natural Language Autoencoderscited by 0
  6. 2026Who Pays More for Safety? Measuring the Disparate Cost of Safety Alignment across Languagescited by 0
  7. 2026The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipelinecited by 0
  8. 2026Retrieval Sensitivity to Identity Signals in Queriescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
  2. 2026Misaligned by Reward: Socially Undesirable Preferences in LLMs
  3. 2026Reward Models Inherit Value Biases from Pretraining
  4. 2026When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models
  5. 2025Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models
  6. 2025Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?
  7. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  8. 2024From Lists to Emojis: How Format Bias Affects Model Alignment