# The Hrunting of AI: Where and How to Improve English Dialectal Fairness

arXiv: https://arxiv.org/abs/2603.15187 · HTML: https://mlsanity.com/paper/2603.15187/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 54 of its 120 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 30 of 54)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Understanding Dataset Difficulty with V-Usable Information](https://mlsanity.com/paper/2110.08420/) (2021)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://mlsanity.com/paper/2405.00332/) (2024)
- [OpenCodeReasoning: Advancing Data Distillation for Competitive Coding](https://mlsanity.com/paper/2504.01943/) (2025)
- [Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?](https://mlsanity.com/paper/2309.07462/) (2023)
- [LLMs Are Few-Shot In-Context Low-Resource Language Learners](https://mlsanity.com/paper/2403.16512/) (2024)
- [Unintended Impacts of LLM Alignment on Global Representation](https://mlsanity.com/paper/2402.15018/) (2024)
- [Can You Trust LLM Judgments? Reliability of LLM-as-a-Judge](https://mlsanity.com/paper/2412.12509/) (2024)
- [Investigating African-American Vernacular English in Transformer-Based Text Generation](https://mlsanity.com/paper/2010.02510/) (2020)
- [DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages](https://mlsanity.com/paper/2403.11009/) (2024)
- [AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs](https://mlsanity.com/paper/2409.11404/) (2024)
- [Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression](https://mlsanity.com/paper/2407.04965/) (2024)
- [Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision](https://mlsanity.com/paper/2012.08012/) (2020)
- [Rejected Dialects: Biases Against African American Language in Reward Models](https://mlsanity.com/paper/2502.12858/) (2025)
- [Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking](https://mlsanity.com/paper/2503.00231/) (2025)
- [A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques](https://mlsanity.com/paper/2406.04879/) (2024)
- [Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning](https://mlsanity.com/paper/2411.18571/) (2024)
- [Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect](https://mlsanity.com/paper/2409.17912/) (2024)
- [NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities](https://mlsanity.com/paper/2505.18383/) (2025)
- [Large Language Models Discriminate Against Speakers of German Dialects](https://mlsanity.com/paper/2509.13835/) (2025)
- [Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties](https://mlsanity.com/paper/2411.10954/) (2024)
- [AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabic](https://mlsanity.com/paper/2412.04193/) (2024)
- [BgGPT 1.0: Extending English-centric LLMs to other languages](https://mlsanity.com/paper/2412.10893/) (2024)
- [SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia](https://mlsanity.com/paper/2503.17485/) (2025)
- [Nile-Chat: Egyptian Language Models for Arabic and Latin Scripts](https://mlsanity.com/paper/2507.04569/) (2025)
- [Languages Still Left Behind: Toward a Better Multilingual Machine Translation Benchmark](https://mlsanity.com/paper/2508.20511/) (2025)
- [Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models](https://mlsanity.com/paper/2409.00358/) (2024)

## Abstract (arXiv)

It is known that large language models (LLMs) underperform in English dialects, and that improving them is difficult due to data scarcity. In this work we investigate how quality and availability impact the feasibility of improving LLMs in this context. For this, we evaluate three rarely-studied English dialects (Yorkshire, Geordie, and Cornish), plus African-American Vernacular English, and West Frisian as control. We find that human-human agreement when determining LLM generation quality directly impacts LLM-as-a-judge performance. That is, LLM-human agreement mimics the human-human agreement pattern, and so do metrics such as accuracy. It is an issue because LLM-human agreement measures an LLM's alignment with the human consensus; and hence raises questions about the feasibility of improving LLM performance in locales where low populations induce low agreement. We also note that fine-tuning does not eradicate, and might amplify, this pattern in English dialects. But also find encouraging signals, such as some LLMs' ability to generate high-quality data, thus enabling scalability. We argue that data must be carefully evaluated to ensure fair and inclusive LLM improvement; and, in the presence of scarcity, new tools are needed to handle the pattern found.