mlsanityData to 7 Oct 2026

Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks

arXiv 2410.11005 ↗First posted 14 Oct 2024cs.CL cs.LGcited by 20

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Language is not monolithic. While benchmarks, including those designed for multiple languages, are often used as proxies to evaluate the performance of Large Language Models (LLMs), they tend to overlook the nuances of within-language variation and thus fail to model the experience of speakers of non-standard dialects. Focusing on African American Vernacular English (AAVE), we present the first study aimed at objectively assessing the fairness and robustness of LLMs in handling dialects across canonical reasoning tasks, including algorithm, math, logic, and integrated reasoning. We introduce ReDial (Reasoning with Dialect Queries), a benchmark containing 1.2K+ parallel query pairs in Standardized English and AAVE. We hire AAVE speakers, including experts with computer science backgrounds, to rewrite seven popular benchmarks, such as HumanEval and GSM8K. With ReDial, we evaluate widely used LLMs, including GPT, Claude, Llama, Mistral, and the Phi model families. Our findings reveal that almost all of these widely used models show significant brittleness and unfairness to queries in AAVE. Our work establishes a systematic and objective framework for analyzing LLM bias in dialectal queries. Moreover, it highlights how mainstream LLMs provide unfair service to dialect speakers in reasoning tasks, laying a critical foundation for future research.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 39 of its 99 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

39 of 99 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024MindMerger: Efficient Boosting LLM Reasoning in non-English Languages27 May
  2. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks5 Jul
  3. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting11 May
  4. 2022Towards Reasoning in Large Language Models: A Survey20 Dec
  5. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021Program Synthesis with Large Language Modelscited by 3,509
  7. 2023Mistral 7Bcited by 3,065
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2024Mixtral of Expertscited by 1,785
  10. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  11. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  12. 2023Textbooks Are All You Needcited by 561
22 more
  1. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  2. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  3. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  4. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  5. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  6. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  7. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  8. 2024Large language models that replace human participants can harmfully misportray and flatten identity groupscited by 137
  9. 2022Holistic Evaluation of Language Modelscited by 115
  10. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  11. 2021Evaluating the Robustness of Neural Language Models to Input Perturbationscited by 87
  12. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Imagescited by 55
  13. 2024Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discriminationcited by 55
  14. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 49
  15. 2022VALUE: Understanding Dialect Disparity in NLUcited by 45
  16. 2022Measuring Reliability of Large Language Models through Semantic Consistencycited by 38
  17. 2024Graph-enhanced Large Language Models in Asynchronous Plan Reasoningcited by 37
  18. 2024Dialect prejudice predicts AI decisions about people's character, employability, and criminalitycited by 35
  19. 2024Stereotype or Personalization? User Identity Biases Chatbot Recommendationscited by 33
  20. 2024AAVENUE: Detecting LLM Biases on NLU Tasks in AAVE via a Novel Benchmarkcited by 11
  21. 2024Evaluating and Mitigating Linguistic Discrimination in Large Language Modelscited by 8
  22. 2024Standard Language Ideology in AI-Generated Languagecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 17.

20 papers.

Sort:

By area:

  1. 2025Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American Englishcited by 8
  2. 2026Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Biascited by 3
  3. 2026Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implicationscited by 2
  4. 2025SLAyiNG: A Diverse and Community-validated Dataset of Queer Slangcited by 1
  5. 2026Digital Linguistic Bias in Spanish: Evidence from Lexical Variation in LLMscited by 1
  6. 2026IDIOLEX: Unified and Continuous Representations for Idiolectal and Stylistic Variationcited by 1
  7. 2025Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responsescited by 0
  8. 2026DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American Englishcited by 0
  9. 2026Evaluating the Usage of African-American Vernacular English in Large Language Modelscited by 0
  10. 2026Can I guess where you are from? Modeling dialectal morphosyntactic similarities in Brazilian Portuguesecited by 0
  11. 2026Individual and Combined Effects of English as a Second Language and Typos on LLM Performancecited by 0
  12. 2026Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Modelscited by 0
8 more
  1. 2026LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steeringcited by 0
  2. 2026AI systems and the reproduction of (standard) language ideologies in World Englishescited by 0
  3. 2026Wiktionary as a Crowdsourced Lexicon for English Dialectscited by 0
  4. 2026CyrillicQA: The Influence of Phonetically Encoded Secret Language on LLM Performancecited by 0
  5. 2026Who Pays More for Safety? Measuring the Disparate Cost of Safety Alignment across Languagescited by 0
  6. 2026The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipelinecited by 0
  7. 2026Not Safe for All: Auditing the Dialect Penalty in Text-to-Image Safety Pipelinescited by 0
  8. 2026Some Dialects Are More Equal Than Others: Non-Prestigious Arabic Dialectal Bias in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks
  2. 2025DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models
  3. 2025LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs
  4. 2023Assessing and Enhancing the Robustness of Large Language Models with Task Structure Variations for Logical Reasoning
  5. 2026MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation
  6. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs
  7. 2026Rethinking the Multilingual Reasoning Gap with Layer Swap
  8. 2022Language Models are Multilingual Chain-of-Thought Reasoners