mlsanityData to 14 Oct 2026

DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages

arXiv 2403.11009 ↗First posted 16 Mar 2024cs.CL cs.AIcited by 50

Not on the map. Why?

Abstract (arXiv)

Language technologies should be judged on their usefulness in real-world use cases. An often overlooked aspect in natural language processing (NLP) research and evaluation is language variation in the form of non-standard dialects or language varieties (hereafter, varieties). Most NLP benchmarks are limited to standard language varieties. To fill this gap, we propose DIALECTBENCH, the first-ever large-scale benchmark for NLP on varieties, which aggregates an extensive set of task-varied variety datasets (10 text-level tasks covering 281 varieties). This allows for a comprehensive evaluation of NLP system performance on different language varieties. We provide substantial evidence of performance disparities between standard and non-standard language varieties, and we also identify language clusters with large performance divergence across tasks. We believe DIALECTBENCH provides a comprehensive view of the current state of NLP for language varieties and one step towards advancing it further. Code/data: https://github.com/ffaisal93/DialectBench

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Builds on 29 of its 64 references that are in the map. First posted on arXiv 16 Mar 2024.

Builds on

29 of 64 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  6. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  7. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  8. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  9. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  10. 2020A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarioscited by 171
  11. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  12. 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 166
17 more
  1. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  2. 2021KLUE: Korean Language Understanding Evaluationcited by 114
  3. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 109
  4. 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 102
  5. 2022Multi-VALUE: A Framework for Cross-Dialectal English NLPcited by 53
  6. 2022VALUE: Understanding Dialect Disparity in NLUcited by 47
  7. 2022ORCA: A Challenging Benchmark for Arabic Language Understandingcited by 47
  8. 2021SD-QA: Spoken Dialectal Question Answering for the Real Worldcited by 38
  9. 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34
  10. 2023Quantifying the Dialect Gap and its Correlates Across Languagescited by 33
  11. 2021A Large-Scale Study of Machine Translation in the Turkic Languagescited by 21
  12. 2023Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languagescited by 19
  13. 2023Language Variety Identification with True Labelscited by 12
  14. 2017Universal Dependencies Parsing for Colloquial Singaporean Englishcited by 10
  15. 2023CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translationcited by 9
  16. 2023CreoleVal: Multilingual Multitask Benchmarks for Creolescited by 9
  17. 2021Finnish Dialect Identification: The Effect of Audio and Textcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 20; 2026: 15.

50 papers.

Sort:

By area:

  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 103
  2. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 28
  3. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 27
  4. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  5. 2024Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Taskscited by 22
  6. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 16
  7. 2024Evaluating Dialect Robustness of Language Models via Conversation Understandingcited by 14
  8. 2024Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varietiescited by 12
  9. 2024Voices Unheard: NLP Resources and Models for Yor`ub'a Regional Dialectscited by 11
  10. 2024AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabiccited by 11
  11. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 9
  12. 2025Tokenization and Representation Biases in Multilingual Models on Dialectal NLP Taskscited by 9
38 more
  1. 2024We're Calling an Intervention: Exploring Fundamental Hurdles in Adapting Language Models to Nonstandard Textcited by 6
  2. 2024Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Modelscited by 6
  3. 2024BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of Englishcited by 5
  4. 2025EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Modelscited by 5
  5. 2024HEALTH-PARIKSHA: Assessing RAG Models for Health Chatbots in Real-World Multilingual Settingscited by 4
  6. 2024Data-Augmentation-Based Dialectal Adaptation for LLMscited by 3
  7. 2025Make Every Letter Count: Building Dialect Variation Dictionaries from Monolingual Corporacited by 3
  8. 2025MEDEQUALQA: Evaluating Biases in LLMs with Counterfactual Reasoningcited by 3
  9. 2025GRDD+: An Extended Greek Dialectal Dataset with Cross-Architecture Fine-tuning Evaluationcited by 3
  10. 2025LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriamacited by 2
  11. 2025We Need to Measure Data Diversity in NLP -- Better and Broadercited by 2
  12. 2025A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systemscited by 2
  13. 2025Lost in Variation? Evaluating NLI Performance in Basque and Spanish Geographical Variantscited by 2
  14. 2025Idiom Understanding as a Tool to Measure the Dialect Gapcited by 2
  15. 2025Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translationcited by 2
  16. 2024Dialetto, ma Quanto Dialetto? Transcribing and Evaluating Dialects on a Continuumcited by 1
  17. 2025Open or Closed LLM for Lesser-Resourced Languages? Lessons from Greekcited by 1
  18. 2025Harnessing Test-time Adaptation for NLU tasks Involving Dialects of Englishcited by 1
  19. 2025Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practicescited by 1
  20. 2026Physical Commonsense Reasoning for Lower-Resourced Languages and Dialects: a Study on Basquecited by 1
  21. 2026DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialectscited by 1
  22. 2026DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptationcited by 1
  23. 2024Exploring Diachronic and Diatopic Changes in Dialect Continua: Tasks, Datasets and Challengescited by 0
  24. 2025Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varietiescited by 0
  25. 2025Benchmarking Sociolinguistic Diversity in Swahili NLP: A Taxonomy-Guided Approachcited by 0
  26. 2025Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responsescited by 0
  27. 2026DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American Englishcited by 0
  28. 2026NLP Privacy Risk Identification in Social Media (NLP-PRISM): A Surveycited by 0
  29. 2026The Hrunting of AI: Where and How to Improve English Dialectal Fairnesscited by 0
  30. 2026A Catalog of Basque Dialectal Resources: Online Collections and Standard-to-Dialectal Adaptationscited by 0
  31. 2026Toward Culturally Grounded Natural Language Processingcited by 0
  32. 2026textitVersteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Languagecited by 0
  33. 2026Saar-Voice: A Multi-Speaker Saarbr"ucken Dialect Speech Corpuscited by 0
  34. 2026Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMscited by 0
  35. 2026Harnessing Linguistic Dissimilarity for Language Generalization on Unseen Low-Resource Varietiescited by 0
  36. 2026Different Perturbations, Different Mechanisms: Understanding Continued Pre-training for Zero-Shot Dialect Robustnesscited by 0
  37. 2026Not Safe for All: Auditing the Dialect Penalty in Text-to-Image Safety Pipelinescited by 0
  38. 2026Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resourcescited by 0