mlsanityData to 14 Oct 2026

Evaluation of African American Language Bias in Natural Language Generation

arXiv 2305.14291 ↗First posted 23 May 2023cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

We evaluate how well LLMs understand African American Language (AAL) in comparison to their performance on White Mainstream English (WME), the encouraged "standard" form of English taught in American classrooms. We measure LLM performance using automatic metrics and human judgments for two tasks: a counterpart generation task, where a model generates AAL (or WME) given WME (or AAL), and a masked span prediction (MSP) task, where models predict a phrase that was removed from their input. Our contributions include: (1) evaluation of six pre-trained, large language models on the two language generation tasks; (2) a novel dataset of AAL text from multiple contexts (social media, hip-hop lyrics, focus groups, and linguistic interviews) with human-annotated counterparts in WME; and (3) documentation of model performance gaps that suggest bias and identification of trends in lack of understanding of AAL features.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 36 of its 90 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

36 of 90 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  9. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  10. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  11. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  12. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
24 more
  1. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  2. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 216
  3. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 209
  4. 2021Societal Biases in Language Generation: Progress and Challengescited by 202
  5. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 196
  6. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  7. 2020Towards Controllable Biases in Language Generationcited by 112
  8. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 96
  9. 2017Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American Englishcited by 71
  10. 2020Demoting Racial Bias in Hate Speech Detectioncited by 62
  11. 2020Investigating African-American Vernacular English in Transformer-Based Text Generationcited by 53
  12. 2022VALUE: Understanding Dialect Disparity in NLUcited by 47
  13. 2021Revealing Persona Biases in Dialogue Systemscited by 33
  14. 2020Learning to Recognize Dialect Featurescited by 29
  15. 2020It's Easier to Translate out of English than into it: Measuring Neural Translation Difficulty by Cross-Mutual Informationcited by 26
  16. 2022Challenges in Measuring Bias via Open-Ended Language Generationcited by 24
  17. 2022Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Taskscited by 17
  18. 2022Corpus-Guided Contrast Sets for Morphosyntactic Feature Detection in Low-Resource English Varietiescited by 6
  19. 2019Unsupervised Neural Dialect Translation with Commonality and Diversity Modelingcited by 5
  20. 2021Subjective Bias in Abstractive Summarizationcited by 5
  21. 2022Disambiguation of morpho-syntactic features of African American English -- the case of habitual becited by 5
  22. 2022Unintended Bias in Language Model-driven Conversational Recommendationcited by 4
  23. 2022Automatic Dialect Density Estimation for African American Englishcited by 4
  24. 2022Identifying and Measuring Token-Level Sentiment Bias in Pre-trained Language Models with Promptscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 12; 2026: 10.

36 papers.

Sort:

By area:

  1. 2024Massively Multi-Cultural Knowledge Acquisition & LM Benchmarkingcited by 63
  2. 2024Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discriminationcited by 56
  3. 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
  4. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 28
  5. 2024CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledgecited by 28
  6. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  7. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  8. 2024Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writerscited by 19
  9. 2024Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varietiescited by 12
  10. 2024AAVENUE: Detecting LLM Biases on NLU Tasks in AAVE via a Novel Benchmarkcited by 11
  11. 2025Guiding LLM Decision-Making with Fairness Reward Modelscited by 9
  12. 2025Language Models Change Facts Based on the Way You Talkcited by 8
24 more
  1. 2024Standard Language Ideology in AI-Generated Languagecited by 5
  2. 2024BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of Englishcited by 5
  3. 2024Towards Effective Discrimination Testing for Generative AIcited by 5
  4. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  5. 2025Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varietiescited by 4
  6. 2026Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profilescited by 4
  7. 2025Data Caricatures: On the Representation of African American Language in Pretraining Corporacited by 3
  8. 2025Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressionscited by 2
  9. 2024MASIVE: Open-Ended Affective State Identification in English and Spanishcited by 1
  10. 2024Training an NLP Scholar at a Small Liberal Arts College: A Backwards Designed Course Proposalcited by 1
  11. 2025Finding A Voice: Exploring the Potential of African American Dialect and Voice Generation for Chatbotscited by 1
  12. 2025SLAyiNG: A Diverse and Community-validated Dataset of Queer Slangcited by 1
  13. 2025Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responsescited by 0
  14. 2025Writing in Symbiosis: Mapping Human Creative Agency in the AI Eracited by 0
  15. 2025Teaching and Critiquing Conceptualization and Operationalization in NLPcited by 0
  16. 2026DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American Englishcited by 0
  17. 2026Far Out: Evaluating Language Models on Slang in Australian and Indian Englishcited by 0
  18. 2026Autoscoring Anticlimax: A Meta-analytic Understanding of AI's Short-answer Shortcomings and Wording Weaknessescited by 0
  19. 2026Whose Story Gets Told? Positionality and Bias in LLM Summaries of Life Narrativescited by 0
  20. 2026An Investigation of Linguistic Biases in LLM-Based Recommendationscited by 0
  21. 2026LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steeringcited by 0
  22. 2026It's How You Ask: Gender-Associated Linguistic Bias in LLMscited by 0
  23. 2026Empath: Tracing Multi-Level Emotion Dynamics in Crisis Counseling Dialoguescited by 0
  24. 2026Retrieval Sensitivity to Identity Signals in Queriescited by 0