# Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks

arXiv: https://arxiv.org/abs/2510.00962 · HTML: https://mlsanity.com/paper/2510.00962/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 10 of its 33 references that are in the map.
- First posted on arXiv 1 Oct 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 10 of 10)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling](https://mlsanity.com/paper/2401.16380/) (2024)
- [Unintended Impacts of LLM Alignment on Global Representation](https://mlsanity.com/paper/2402.15018/) (2024)
- [Multi-VALUE: A Framework for Cross-Dialectal English NLP](https://mlsanity.com/paper/2212.08011/) (2022)
- [VALUE: Understanding Dialect Disparity in NLU](https://mlsanity.com/paper/2204.03031/) (2022)
- [DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules](https://mlsanity.com/paper/2305.13406/) (2023)

## Built on by (most cited first, 5 of 5)

- [Physical Commonsense Reasoning for Lower-Resourced Languages and Dialects: a Study on Basque](https://mlsanity.com/paper/2602.14812/) (2026)
- [P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs](https://mlsanity.com/paper/2606.16753/) (2026)
- [The Hrunting of AI: Where and How to Improve English Dialectal Fairness](https://mlsanity.com/paper/2603.15187/) (2026)
- [Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Once](https://mlsanity.com/paper/2604.01504/) (2026)
- [Some Dialects Are More Equal Than Others: Non-Prestigious Arabic Dialectal Bias in LLMs](https://mlsanity.com/paper/2609.23955/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the effects of typifying "standard" American English language questions as non-"standard" dialectal variants on multiple choice question answering tasks and find up to a 20% reduction in accuracy. Additionally, we investigate the grammatical basis of under-performance in non-"standard" English questions. We find that individual grammatical rules have varied effects on performance, but some are more consequential than others: three specific grammar rules (existential "it", zero copula, and y'all) can explain the majority of performance degradation observed in multiple dialects. We call for future work to investigate bias mitigation methods focused on individual, high-impact grammatical structures.