# Natural Language Processing for Dialects of a Language: A Survey

arXiv: https://arxiv.org/abs/2401.05632 · HTML: https://mlsanity.com/paper/2401.05632/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 247 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 30 of 59)

- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Demographic Dialectal Variation in Social Media: A Case Study of African-American English](https://mlsanity.com/paper/1608.08868/) (2016)
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://mlsanity.com/paper/2305.13707/) (2023)
- [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://mlsanity.com/paper/2103.06678/) (2021)
- [Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT model](https://mlsanity.com/paper/2008.06460/) (2020)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [Multi-VALUE: A Framework for Cross-Dialectal English NLP](https://mlsanity.com/paper/2212.08011/) (2022)
- [NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2010.11334/) (2020)
- [VALUE: Understanding Dialect Disparity in NLU](https://mlsanity.com/paper/2204.03031/) (2022)
- [A Panoramic Survey of Natural Language Processing in the Arab World](https://mlsanity.com/paper/2011.12631/) (2020)
- [From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding](https://mlsanity.com/paper/2105.07316/) (2021)
- [Automatic Language Identification in Texts: A Survey](https://mlsanity.com/paper/1804.08186/) (2018)
- [Quantifying the Dialect Gap and its Correlates Across Languages](https://mlsanity.com/paper/2310.15135/) (2023)
- [NADI 2021: The Second Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2103.08466/) (2021)
- [Learning to Recognize Dialect Features](https://mlsanity.com/paper/2010.12707/) (2020)
- [NADI 2023: The Fourth Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2310.16117/) (2023)
- [ALDi: Quantifying the Arabic Level of Dialectness of Text](https://mlsanity.com/paper/2310.13747/) (2023)
- [NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2210.09582/) (2022)
- [Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding](https://mlsanity.com/paper/2004.14870/) (2020)
- [MultiSpider: Towards Benchmarking Multilingual Text-to-SQL Semantic Parsing](https://mlsanity.com/paper/2212.13492/) (2022)
- [SocialDial: A Benchmark for Socially-Aware Dialogue Systems](https://mlsanity.com/paper/2304.12026/) (2023)
- [Machine Translation into Low-resource Language Varieties](https://mlsanity.com/paper/2106.06797/) (2021)
- [Improving Zero-shot Cross-lingual Transfer between Closely Related Languages by injecting Character-level Noise](https://mlsanity.com/paper/2109.06772/) (2021)
- [Does Manipulating Tokenization Aid Cross-Lingual Transfer? A Study on POS Tagging for Non-Standardized Languages](https://mlsanity.com/paper/2304.10158/) (2023)
- [FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation](https://mlsanity.com/paper/2210.00193/) (2022)
- [STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions](https://mlsanity.com/paper/2305.18855/) (2023)
- [A Large Scale Corpus of Gulf Arabic](https://mlsanity.com/paper/1609.02960/) (2016)
- [Dialect Diversity in Text Summarization on Twitter](https://mlsanity.com/paper/2007.07860/) (2020)
- [Disentangling Representations of Text by Masking Transformers](https://mlsanity.com/paper/2104.07155/) (2021)
- [TADA: Task-Agnostic Dialect Adapters for English](https://mlsanity.com/paper/2305.16651/) (2023)

## Built on by (most cited first, 30 of 40)

- [Beyond Perplexity: Multi-dimensional Safety Evaluation of LLM Compression](https://mlsanity.com/paper/2407.04965/) (2024)
- [RENOVI: A Benchmark Towards Remediating Norm Violations in Socio-Cultural Conversations](https://mlsanity.com/paper/2402.11178/) (2024)
- [Evaluating Dialect Robustness of Language Models via Conversation Understanding](https://mlsanity.com/paper/2405.05688/) (2024)
- [Voices Unheard: NLP Resources and Models for Yor`ub'a Regional Dialects](https://mlsanity.com/paper/2406.19564/) (2024)
- [AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabic](https://mlsanity.com/paper/2412.04193/) (2024)
- [We're Calling an Intervention: Exploring Fundamental Hurdles in Adapting Language Models to Nonstandard Text](https://mlsanity.com/paper/2404.07304/) (2024)
- [Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models](https://mlsanity.com/paper/2409.00358/) (2024)
- [Standard Language Ideology in AI-Generated Language](https://mlsanity.com/paper/2406.08726/) (2024)
- [BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English](https://mlsanity.com/paper/2412.04726/) (2024)
- [Enhancing Portuguese Variety Identification with Cross-Domain Approaches](https://mlsanity.com/paper/2502.14394/) (2025)
- [Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties](https://mlsanity.com/paper/2505.20875/) (2025)
- [An End-to-End Real-World Camera Imaging Pipeline](https://mlsanity.com/paper/2411.10773/) (2024)
- [MassTool: A Multi-Task Search-Based Tool Retrieval Framework for Large Language Models](https://mlsanity.com/paper/2507.00487/) (2025)
- [Make Every Letter Count: Building Dialect Variation Dictionaries from Monolingual Corpora](https://mlsanity.com/paper/2509.17855/) (2025)
- [How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipeline](https://mlsanity.com/paper/2604.04204/) (2026)
- [Dialect Normalization using Large Language Models and Morphological Rules](https://mlsanity.com/paper/2506.08907/) (2025)
- [Lost in Variation? Evaluating NLI Performance in Basque and Spanish Geographical Variants](https://mlsanity.com/paper/2506.15239/) (2025)
- [Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects](https://mlsanity.com/paper/2510.07890/) (2025)
- [DialUp! Modeling the Language Continuum by Adapting Models to Dialects and Dialects to Models](https://mlsanity.com/paper/2501.16581/) (2025)
- [Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English](https://mlsanity.com/paper/2503.12858/) (2025)
- [Improving Low-Resource Dialect Classification Using Retrieval-based Voice Conversion](https://mlsanity.com/paper/2507.03641/) (2025)
- [NLD-LLM: A systematic framework for evaluating small language transformer models on natural language description](https://mlsanity.com/paper/2510.05139/) (2025)
- [DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects](https://mlsanity.com/paper/2604.05318/) (2026)
- [DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation](https://mlsanity.com/paper/2607.07669/) (2026)
- [Let's Negotiate! A Survey of Negotiation Dialogue Systems](https://mlsanity.com/paper/2402.01097/) (2024)
- [Exploring Diachronic and Diatopic Changes in Dialect Continua: Tasks, Datasets and Challenges](https://mlsanity.com/paper/2407.04010/) (2024)
- [Unification of Balti and trans-border sister dialects in the essence of LLMs and AI Technology](https://mlsanity.com/paper/2411.13409/) (2024)
- [Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varieties](https://mlsanity.com/paper/2502.07058/) (2025)
- [GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models](https://mlsanity.com/paper/2504.04155/) (2025)
- [Splits! Flexible Sociocultural Linguistic Investigation at Scale](https://mlsanity.com/paper/2504.04640/) (2025)

## Abstract (arXiv)

State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language. Motivated by the performance degradation of NLP models for dialectal datasets and its implications for the equity of language technologies, we survey past research in NLP for dialects in terms of datasets, and approaches. We describe a wide range of NLP tasks in terms of two categories: natural language understanding (NLU) (for tasks such as dialect classification, sentiment analysis, parsing, and NLU benchmarks) and natural language generation (NLG) (for summarisation, machine translation, and dialogue systems). The survey is also broad in its coverage of languages which include English, Arabic, German, among others. We observe that past work in NLP concerning dialects goes deeper than mere dialect classification, and extends to several NLU and NLG tasks. For these tasks, we describe classical machine learning using statistical models, along with the recent deep learning-based approaches based on pre-trained language models. We expect that this survey will be useful to NLP researchers interested in building equitable language technologies by rethinking LLM benchmarks and model architectures.