mlsanityData to 14 Oct 2026

Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks

arXiv 2510.00962 ↗First posted 1 Oct 2025cs.CLcited by 5

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the effects of typifying "standard" American English language questions as non-"standard" dialectal variants on multiple choice question answering tasks and find up to a 20% reduction in accuracy. Additionally, we investigate the grammatical basis of under-performance in non-"standard" English questions. We find that individual grammatical rules have varied effects on performance, but some are more consequential than others: three specific grammar rules (existential "it", zero copula, and y'all) can explain the majority of performance degradation observed in multiple dialects. We call for future work to investigate bias mitigation methods focused on individual, high-impact grammatical structures.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 10 of its 33 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

10 of 33 references.

References · 10

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Mistral 7Bcited by 3,164
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  5. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  6. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  7. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  8. 2022Multi-VALUE: A Framework for Cross-Dialectal English NLPcited by 53
  9. 2022VALUE: Understanding Dialect Disparity in NLUcited by 47
  10. 2023DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rulescited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Physical Commonsense Reasoning for Lower-Resourced Languages and Dialects: a Study on Basquecited by 1
  2. 2026P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMscited by 1
  3. 2026The Hrunting of AI: Where and How to Improve English Dialectal Fairnesscited by 0
  4. 2026Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Oncecited by 0
  5. 2026Some Dialects Are More Equal Than Others: Non-Prestigious Arabic Dialectal Bias in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Dialectic Pipeline for Improving LLM Robustness
  2. 2025None of the Above, Less of the Right: Parallel Patterns between Humans and LLMs on Multi-Choice Questions Answering
  3. 2025Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?
  4. 2025Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American English
  5. 2026Exploring the Meta-level Reasoning of Large Language Models via a Tool-based Multi-hop Tabular Question Answering Task
  6. 2024Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks
  7. 2023NLPBench: Evaluating Large Language Models on Solving NLP Problems
  8. 2025DialectalArabicMMLU: Benchmarking Dialectal Capabilities in Arabic and Multilingual Language Models