mlsanityData to 14 Oct 2026

Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination

arXiv 2406.08818 ↗First posted 13 Jun 2024cs.CL cs.CYcited by 56

Not on the map. Why?

Abstract (arXiv)

We present a large-scale study of linguistic bias exhibited by ChatGPT covering ten dialects of English (Standard American English, Standard British English, and eight widely spoken non-"standard" varieties from around the world). We prompted GPT-3.5 Turbo and GPT-4 with text by native speakers of each variety and analyzed the responses via detailed linguistic feature annotation and native speaker evaluation. We find that the models default to "standard" varieties of English; based on evaluation by native speakers, we also find that model responses to non-"standard" varieties consistently exhibit a range of issues: stereotyping (19% worse than for "standard" varieties), demeaning content (25% worse), lack of comprehension (9% worse), and condescending responses (15% worse). We also find that if these models are asked to imitate the writing style of prompts in non-"standard" varieties, they produce text that exhibits lower comprehension of the input and is especially prone to stereotyping. GPT-4 improves on GPT-3.5 in terms of comprehension, warmth, and friendliness, but also exhibits a marked increase in stereotyping (+18%). The results indicate that GPT-3.5 Turbo and GPT-4 can perpetuate linguistic discrimination toward speakers of non-"standard" varieties.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Builds on 6 of its 53 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

6 of 53 references.

References · 6

  1. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  2. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  3. 2022Multi-VALUE: A Framework for Cross-Dialectal English NLPcited by 53
  4. 2023Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languagescited by 51
  5. 2024Dialect prejudice predicts AI decisions about people's character, employability, and criminalitycited by 37
  6. 2023Evaluation of African American Language Bias in Natural Language Generationcited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 22; 2026: 28.

56 papers.

Sort:

By area:

  1. 2024Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Taskscited by 22
  2. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  3. 2025What's Not Said Still Hurts: A Description-Based Evaluation Framework for Measuring Social Bias in LLMscited by 8
  4. 2024Ads that Talk Back: Implications and Perceptions of Injecting Personalized Advertising into LLM Chatbotscited by 7
  5. 2025Unsupervised Concept Vector Extraction for Bias Control in LLMscited by 7
  6. 2025Invisible Filters: Cultural Bias in Hiring Evaluations Using Large Language Modelscited by 6
  7. 2025EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Modelscited by 5
  8. 2025Fragile Preferences: A Deep Dive Into Order Effects in Large Language Modelscited by 5
  9. 2025Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varietiescited by 4
  10. 2026Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Biascited by 4
  11. 2024Performance in a dialectal profiling task of LLMs for varieties of Brazilian Portuguesecited by 3
  12. 2025DiffSampling: Enhancing Diversity and Accuracy in Neural Text Generationcited by 3
38 more
  1. 2025Data Caricatures: On the Representation of African American Language in Pretraining Corporacited by 3
  2. 2025MEDEQUALQA: Evaluating Biases in LLMs with Counterfactual Reasoningcited by 3
  3. 2026How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipelinecited by 3
  4. 2024Data-Centric AI Governance: Addressing the Limitations of Model-Focused Policiescited by 2
  5. 2025Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressionscited by 2
  6. 2025Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translationcited by 2
  7. 2026IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variationcited by 2
  8. 2026Topics as Proxies for Sociodemographics: How Conversational Context Affects LLM Answerscited by 2
  9. 2025Finding A Voice: Exploring the Potential of African American Dialect and Voice Generation for Chatbotscited by 1
  10. 2025Harnessing Test-time Adaptation for NLU tasks Involving Dialects of Englishcited by 1
  11. 2025Ethics and Persuasion in Reinforcement Learning from Human Feedback: A Procedural Rhetorical Approachcited by 1
  12. 2025I Think, Therefore I Am Under-Qualified? A Benchmark for Evaluating Linguistic Shibboleth Detection in LLM Hiring Evaluationscited by 1
  13. 2026DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptationcited by 1
  14. 2024The Language of Weather: Social Media Reactions to Weather Accounting for Climatic and Linguistic Baselinescited by 0
  15. 2024Diversidade lingu'istica e inclus~ao digital: desafios para uma ia brasileiracited by 0
  16. 2025Whispers of Many Shores: Cultural Alignment through Collaborative Cultural Expertisecited by 0
  17. 2025Comparative Analysis of Large Language Models for the Machine-Assisted Resolution of User Intentionscited by 0
  18. 2025Automated Coding of Communication Data Using ChatGPT: Consistency Across Subgroupscited by 0
  19. 2025CLARITY: Contextual Linguistic Adaptation and Accent Retrieval for Dual-Bias Mitigation in Text-to-Speech Generationcited by 0
  20. 2025Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responsescited by 0
  21. 2025When the Gold Standard Isn't Necessarily Standard: Challenges of Evaluating the Translation of User-Generated Contentcited by 0
  22. 2026Can AI mediation improve democratic deliberation?cited by 0
  23. 2026IndRegBias: A Dataset for Studying Indian Regional Biases in English and Code-Mixed Social Media Commentscited by 0
  24. 2026DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American Englishcited by 0
  25. 2026NLP Privacy Risk Identification in Social Media (NLP-PRISM): A Surveycited by 0
  26. 2026Evaluating the Usage of African-American Vernacular English in Large Language Modelscited by 0
  27. 2026Analysis Of Linguistic Stereotypes in Single and Multi-Agent Generative AI Architecturescited by 0
  28. 2026Can I guess where you are from? Modeling dialectal morphosyntactic similarities in Brazilian Portuguesecited by 0
  29. 2026Benchmarking Bengali Dialectal Bias: A Multi-Stage Framework Integrating RAG-Based Translation and Human-Augmented RLAIFcited by 0
  30. 2026An Investigation of Linguistic Biases in LLM-Based Recommendationscited by 0
  31. 2026From Standard English to Singlish: A Retrieval-Augmented Approach for Code-Switched Creole Generation in Large Language Modelscited by 0
  32. 2026Side-by-side Comparison Amplifies Dialect Bias in Language Modelscited by 0
  33. 2026Beyond Third-Person Audits: Situated Interaction Auditing for User-Centered LLM Bias Researchcited by 0
  34. 2026LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steeringcited by 0
  35. 2026Probing Latent Colombian Identity Inferences in Qwen2.5-7B with Natural Language Autoencoderscited by 0
  36. 2026Generative AI and linguistic diversity in academic writing and publishing: Perspectives from World Englishescited by 0
  37. 2026AI systems and the reproduction of (standard) language ideologies in World Englishescited by 0
  38. 2026It's How You Ask: Gender-Associated Linguistic Bias in LLMscited by 0

50 most cited of 56 All 56 (JSON)