mlsanityData to 14 Oct 2026

Quantifying the Dialect Gap and its Correlates Across Languages

arXiv 2310.15135 ↗First posted 23 Oct 2023cs.CL cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Historically, researchers and consumers have noticed a decrease in quality when applying NLP tools to minority variants of languages (i.e. Puerto Rican Spanish or Swiss German), but studies exploring this have been limited to a select few languages. Additionally, past studies have mainly been conducted in a monolingual context, so cross-linguistic trends have not been identified and tied to external factors. In this work, we conduct a comprehensive evaluation of the most influential, state-of-the-art large language models (LLMs) across two high-use applications, machine translation and automatic speech recognition, to assess their functionality on the regional dialects of several high- and low-resource languages. Additionally, we analyze how the regional dialect gap is correlated with economic, social, and linguistic factors. The impact of training data, including related factors like dataset size and its construction procedure, is shown to be significant but not consistent across models or languages, meaning a one-size-fits-all approach cannot be taken in solving the dialect gap. This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 31 of its 71 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

31 of 71 references.

References · 31

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  6. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  7. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  8. 2019Massively Multilingual Neural Machine Translationcited by 246
  9. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  10. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  11. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 144
  12. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 128
19 more
  1. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
  2. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 66
  3. 2018Modeling Language Variation and Universals: A Survey on Typological Linguistics for Natural Language Processingcited by 63
  4. 2021Beyond Fair Pay: Ethical Implications of NLP Crowdsourcingcited by 53
  5. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  6. 2021Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understandingcited by 42
  7. 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34
  8. 2022Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifoldcited by 29
  9. 2023Zero-shot cross-lingual transfer language selection using linguistic similaritycited by 28
  10. 2022How can NLP Help Revitalize Endangered Languages? A Case Study and Roadmap for the Cherokee Languagecited by 19
  11. 2021How Good Is NLP? A Sober Look at NLP Tasks through the Lens of Social Impactcited by 18
  12. 2022FRMT: A Benchmark for Few-Shot Region-Aware Machine Translationcited by 18
  13. 2020Facebook AI's WMT20 News Translation Task Submissioncited by 17
  14. 2021Using Radio Archives for Low-Resource Speech Recognition: Towards an Intelligent Virtual Assistant for Illiterate Userscited by 16
  15. 2021Semantic-WER: A Unified Metric for the Evaluation of ASR Transcript for End Usabilitycited by 14
  16. 2023Identifying the Correlation Between Language Distance and Cross-Lingual Transfer in a Multilingual Representation Spacecited by 12
  17. 2022Original or Translated? A Causal Analysis of the Impact of Translationese on Machine Translation Performancecited by 11
  18. 2021Representations of Language Varieties Are Reliable Given Corpus Similarity Measurescited by 10
  19. 2023Along the Margins: Marginalized Communities' Ethical Concerns about Social Platformscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 12; 2026: 10.

33 papers.

Sort:

By area:

  1. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 50
  2. 2024Natural Language Processing for Dialects of a Language: A Surveycited by 40
  3. 2024Stereotype or Personalization? User Identity Biases Chatbot Recommendationscited by 34
  4. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  5. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  6. 2024Voices Unheard: NLP Resources and Models for Yor`ub'a Regional Dialectscited by 11
  7. 2025Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Taskscited by 9
  8. 2025TiEBe: Tracking Language Model Recall of Notable Worldwide Events Through Timecited by 8
  9. 2024Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalizationcited by 7
  10. 2025A Multi-Dialectal Dataset for German Dialect ASR and Dialect-to-Standard Speech Translationcited by 7
  11. 2024We're Calling an Intervention: Exploring Fundamental Hurdles in Adapting Language Models to Nonstandard Textcited by 6
  12. 2024Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varietiescited by 4
21 more
  1. 2025Designing Speech Technologies for Australian Aboriginal English: Opportunities, Risks and Participationcited by 3
  2. 2025When Alignment Hurts: Decoupling Representational Spaces in Multilingual Modelscited by 3
  3. 2025Idiom Understanding as a Tool to Measure the Dialect Gapcited by 2
  4. 2025Low-Resource Dialect Adaptation of Large Language Models: A French Dialect Case-Studycited by 2
  5. 2025Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translationcited by 2
  6. 2026Dialect Matters: Cross-Lingual ASR Transfer for Low-Resource Indic Language Varietiescited by 2
  7. 2024Disentangling Dialect from Social Bias via Multitask Learning to Improve Fairnesscited by 1
  8. 2024Dialetto, ma Quanto Dialetto? Transcribing and Evaluating Dialects on a Continuumcited by 1
  9. 2025Vuyko Mistral: Adapting LLMs for Low-Resource Dialectal Translationcited by 1
  10. 2026Digital Linguistic Bias in Spanish: Evidence from Lexical Variation in LLMscited by 1
  11. 2026DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialectscited by 1
  12. 2024Unification of Balti and trans-border sister dialects in the essence of LLMs and AI Technologycited by 0
  13. 2025Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varietiescited by 0
  14. 2025Benchmarking Sociolinguistic Diversity in Swahili NLP: A Taxonomy-Guided Approachcited by 0
  15. 2026Mi'ci Princ -- A Little Boy Teaching Speech Technologies the Chakavian Dialectcited by 0
  16. 2026Meenz bleibt Meenz, but Large Language Models Do Not Speak Its Dialectcited by 0
  17. 2026Convex Low-resource Accent-Robust Language Detection in Speech Recognitioncited by 0
  18. 2026Probing Latent Colombian Identity Inferences in Qwen2.5-7B with Natural Language Autoencoderscited by 0
  19. 2026Different Perturbations, Different Mechanisms: Understanding Continued Pre-training for Zero-Shot Dialect Robustnesscited by 0
  20. 2026CyrillicQA: The Influence of Phonetically Encoded Secret Language on LLM Performancecited by 0
  21. 2026The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipelinecited by 0