# Quantifying the Dialect Gap and its Correlates Across Languages

arXiv: https://arxiv.org/abs/2310.15135 · HTML: https://mlsanity.com/paper/2310.15135/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 71 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 30 of 31)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [Social Biases in NLP Models as Barriers for Persons with Disabilities](https://mlsanity.com/paper/2005.00813/) (2020)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT](https://mlsanity.com/paper/2010.06354/) (2020)
- [Systematic Inequalities in Language Technology Performance across the World's Languages](https://mlsanity.com/paper/2110.06733/) (2021)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processing](https://mlsanity.com/paper/1807.00914/) (2018)
- [Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing](https://mlsanity.com/paper/2104.10097/) (2021)
- [Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLP](https://mlsanity.com/paper/2109.06598/) (2021)
- [Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding](https://mlsanity.com/paper/2104.02138/) (2021)
- [XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages](https://mlsanity.com/paper/2305.11938/) (2023)
- [Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold](https://mlsanity.com/paper/2206.09755/) (2022)
- [Zero-shot cross-lingual transfer language selection using linguistic similarity](https://mlsanity.com/paper/2301.13720/) (2023)
- [How can NLP Help Revitalize Endangered Languages? A Case Study and Roadmap for the Cherokee Language](https://mlsanity.com/paper/2204.11909/) (2022)
- [How Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social Impact](https://mlsanity.com/paper/2106.02359/) (2021)
- [FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation](https://mlsanity.com/paper/2210.00193/) (2022)
- [Facebook AI's WMT20 News Translation Task Submission](https://mlsanity.com/paper/2011.08298/) (2020)
- [Using Radio Archives for Low-Resource Speech Recognition: Towards an Intelligent Virtual Assistant for Illiterate Users](https://mlsanity.com/paper/2104.13083/) (2021)
- [Semantic-WER: A Unified Metric for the Evaluation of ASR Transcript for End Usability](https://mlsanity.com/paper/2106.02016/) (2021)
- [Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Space](https://mlsanity.com/paper/2305.02151/) (2023)
- [Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performance](https://mlsanity.com/paper/2205.02293/) (2022)
- [Representations of Language Varieties Are Reliable Given Corpus Similarity Measures](https://mlsanity.com/paper/2104.01294/) (2021)

## Built on by (most cited first, 30 of 33)

- [DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages](https://mlsanity.com/paper/2403.11009/) (2024)
- [Natural Language Processing for Dialects of a Language: A Survey](https://mlsanity.com/paper/2401.05632/) (2024)
- [Stereotype or Personalization? User Identity Biases Chatbot Recommendations](https://mlsanity.com/paper/2410.05613/) (2024)
- [Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art](https://mlsanity.com/paper/2406.03930/) (2024)
- [Rejected Dialects: Biases Against African American Language in Reward Models](https://mlsanity.com/paper/2502.12858/) (2025)
- [Voices Unheard: NLP Resources and Models for Yor`ub'a Regional Dialects](https://mlsanity.com/paper/2406.19564/) (2024)
- [Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Tasks](https://mlsanity.com/paper/2509.20045/) (2025)
- [TiEBe: Tracking Language Model Recall of Notable Worldwide Events Through Time](https://mlsanity.com/paper/2501.07482/) (2025)
- [Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization](https://mlsanity.com/paper/2406.13718/) (2024)
- [A Multi-Dialectal Dataset for German Dialect ASR and Dialect-to-Standard Speech Translation](https://mlsanity.com/paper/2506.02894/) (2025)
- [We're Calling an Intervention: Exploring Fundamental Hurdles in Adapting Language Models to Nonstandard Text](https://mlsanity.com/paper/2404.07304/) (2024)
- [Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties](https://mlsanity.com/paper/2402.02078/) (2024)
- [Designing Speech Technologies for Australian Aboriginal English: Opportunities, Risks and Participation](https://mlsanity.com/paper/2503.03186/) (2025)
- [When Alignment Hurts: Decoupling Representational Spaces in Multilingual Models](https://mlsanity.com/paper/2508.12803/) (2025)
- [Idiom Understanding as a Tool to Measure the Dialect Gap](https://mlsanity.com/paper/2510.05026/) (2025)
- [Low-Resource Dialect Adaptation of Large Language Models: A French Dialect Case-Study](https://mlsanity.com/paper/2510.22747/) (2025)
- [Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translation](https://mlsanity.com/paper/2511.06680/) (2025)
- [Dialect Matters: Cross-Lingual ASR Transfer for Low-Resource Indic Language Varieties](https://mlsanity.com/paper/2601.04373/) (2026)
- [Disentangling Dialect from Social Bias via Multitask Learning to Improve Fairness](https://mlsanity.com/paper/2406.09977/) (2024)
- [Dialetto, ma Quanto Dialetto? Transcribing and Evaluating Dialects on a Continuum](https://mlsanity.com/paper/2410.14589/) (2024)
- [Vuyko Mistral: Adapting LLMs for Low-Resource Dialectal Translation](https://mlsanity.com/paper/2506.07617/) (2025)
- [Digital Linguistic Bias in Spanish: Evidence from Lexical Variation in LLMs](https://mlsanity.com/paper/2602.09346/) (2026)
- [DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects](https://mlsanity.com/paper/2604.05318/) (2026)
- [Unification of Balti and trans-border sister dialects in the essence of LLMs and AI Technology](https://mlsanity.com/paper/2411.13409/) (2024)
- [Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varieties](https://mlsanity.com/paper/2502.07058/) (2025)
- [Benchmarking Sociolinguistic Diversity in Swahili NLP: A Taxonomy-Guided Approach](https://mlsanity.com/paper/2508.14051/) (2025)
- [Mi'ci Princ -- A Little Boy Teaching Speech Technologies the Chakavian Dialect](https://mlsanity.com/paper/2602.03245/) (2026)
- [Meenz bleibt Meenz, but Large Language Models Do Not Speak Its Dialect](https://mlsanity.com/paper/2602.16852/) (2026)
- [Convex Low-resource Accent-Robust Language Detection in Speech Recognition](https://mlsanity.com/paper/2605.23235/) (2026)
- [Probing Latent Colombian Identity Inferences in Qwen2.5-7B with Natural Language Autoencoders](https://mlsanity.com/paper/2607.21774/) (2026)

## Abstract (arXiv)

Historically, researchers and consumers have noticed a decrease in quality when applying NLP tools to minority variants of languages (i.e. Puerto Rican Spanish or Swiss German), but studies exploring this have been limited to a select few languages. Additionally, past studies have mainly been conducted in a monolingual context, so cross-linguistic trends have not been identified and tied to external factors. In this work, we conduct a comprehensive evaluation of the most influential, state-of-the-art large language models (LLMs) across two high-use applications, machine translation and automatic speech recognition, to assess their functionality on the regional dialects of several high- and low-resource languages. Additionally, we analyze how the regional dialect gap is correlated with economic, social, and linguistic factors. The impact of training data, including related factors like dataset size and its construction procedure, is shown to be significant but not consistent across models or languages, meaning a one-size-fits-all approach cannot be taken in solving the dialect gap. This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.