# Multi-VALUE: A Framework for Cross-Dialectal English NLP

arXiv: https://arxiv.org/abs/2212.08011 · HTML: https://mlsanity.com/paper/2212.08011/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 104 references that are in the map.
- First posted on arXiv 15 Dec 2022.

## Builds on (most cited first, 30 of 40)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Six Challenges for Neural Machine Translation](https://mlsanity.com/paper/1706.03872/) (2017)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [Characterizing and Avoiding Negative Transfer](https://mlsanity.com/paper/1811.09751/) (2018)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Demographic Dialectal Variation in Social Media: A Case Study of African-American English](https://mlsanity.com/paper/1608.08868/) (2016)
- [Reformulating Unsupervised Style Transfer as Paraphrase Generation](https://mlsanity.com/paper/2010.05700/) (2020)
- [Choosing Transfer Languages for Cross-Lingual Learning](https://mlsanity.com/paper/1905.12688/) (2019)
- [Systematic Inequalities in Language Technology Performance across the World's Languages](https://mlsanity.com/paper/2110.06733/) (2021)
- [Challenges in Automated Debiasing for Toxic Language Detection](https://mlsanity.com/paper/2102.00086/) (2021)
- [Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT model](https://mlsanity.com/paper/2008.06460/) (2020)
- [Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American English](https://mlsanity.com/paper/1707.00061/) (2017)
- [Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings](https://mlsanity.com/paper/1904.01596/) (2019)
- [Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection](https://mlsanity.com/paper/2201.10474/) (2022)
- [Perturbation Augmentation for Fairer NLP](https://mlsanity.com/paper/2205.12586/) (2022)
- [The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation](https://mlsanity.com/paper/2110.03036/) (2021)
- [VALUE: Understanding Dialect Disparity in NLU](https://mlsanity.com/paper/2204.03031/) (2022)
- [SD-QA: Spoken Dialectal Question Answering for the Real World](https://mlsanity.com/paper/2109.12072/) (2021)
- [Natural Language Inference over Interaction Space](https://mlsanity.com/paper/1709.04348/) (2017)
- [Anaphora and Coreference Resolution: A Review](https://mlsanity.com/paper/1805.11824/) (2018)

## Built on by (most cited first, 30 of 53)

- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [Unintended Impacts of LLM Alignment on Global Representation](https://mlsanity.com/paper/2402.15018/) (2024)
- [Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discrimination](https://mlsanity.com/paper/2406.08818/) (2024)
- [DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages](https://mlsanity.com/paper/2403.11009/) (2024)
- [Natural Language Processing for Dialects of a Language: A Survey](https://mlsanity.com/paper/2401.05632/) (2024)
- [Culture is Not Trivia: Sociocultural Theory for Cultural NLP](https://mlsanity.com/paper/2502.12057/) (2025)
- [Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art](https://mlsanity.com/paper/2406.03930/) (2024)
- [A Material Lens on Coloniality in NLP](https://mlsanity.com/paper/2311.08391/) (2023)
- [Position: Key Claims in LLM Research Have a Long Tail of Footnotes](https://mlsanity.com/paper/2308.07120/) (2023)
- [A Survey of Corpora for Germanic Low-Resource Languages and Dialects](https://mlsanity.com/paper/2304.09805/) (2023)
- [TADA: Task-Agnostic Dialect Adapters for English](https://mlsanity.com/paper/2305.16651/) (2023)
- [Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classifications](https://mlsanity.com/paper/2310.14607/) (2023)
- [Evaluating Dialect Robustness of Language Models via Conversation Understanding](https://mlsanity.com/paper/2405.05688/) (2024)
- [DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules](https://mlsanity.com/paper/2305.13406/) (2023)
- [Tokenization is Sensitive to Language Variation](https://mlsanity.com/paper/2502.15343/) (2025)
- [Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties](https://mlsanity.com/paper/2411.10954/) (2024)
- [Voices Unheard: NLP Resources and Models for Yor`ub'a Regional Dialects](https://mlsanity.com/paper/2406.19564/) (2024)
- [Out of Style: RAG's Fragility to Linguistic Variation](https://mlsanity.com/paper/2504.08231/) (2025)
- [Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models](https://mlsanity.com/paper/2409.00358/) (2024)
- [Language and Speech Technology for Central Kurdish Varieties](https://mlsanity.com/paper/2403.01983/) (2024)
- [Native Design Bias: Studying the Impact of English Nativeness on Language Model Performance](https://mlsanity.com/paper/2406.17385/) (2024)
- [BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English](https://mlsanity.com/paper/2412.04726/) (2024)
- [Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks](https://mlsanity.com/paper/2510.00962/) (2025)
- [Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties](https://mlsanity.com/paper/2402.02078/) (2024)
- [Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varieties](https://mlsanity.com/paper/2505.20875/) (2025)
- [When Claims Evolve: Evaluating and Enhancing the Robustness of Embedding Models Against Misinformation Edits](https://mlsanity.com/paper/2503.03417/) (2025)
- [DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation](https://mlsanity.com/paper/2510.14949/) (2025)
- [How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipeline](https://mlsanity.com/paper/2604.04204/) (2026)
- [Analysis of LLM as a grammatical feature tagger for African American English](https://mlsanity.com/paper/2502.06004/) (2025)
- [A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systems](https://mlsanity.com/paper/2506.02998/) (2025)

## Abstract (arXiv)

Dialect differences caused by regional, social, and economic factors cause performance discrepancies for many groups of language technology users. Inclusive and equitable language technology must critically be dialect invariant, meaning that performance remains constant over dialectal shifts. Current systems often fall short of this ideal since they are designed and tested on a single dialect: Standard American English (SAE). We introduce a suite of resources for evaluating and achieving English dialect invariance. The resource is called Multi-VALUE, a controllable rule-based translation system spanning 50 English dialects and 189 unique linguistic features. Multi-VALUE maps SAE to synthetic forms of each dialect. First, we use this system to stress tests question answering, machine translation, and semantic parsing. Stress tests reveal significant performance disparities for leading models on non-standard dialects. Second, we use this system as a data augmentation technique to improve the dialect robustness of existing systems. Finally, we partner with native speakers of Chicano and Indian English to release new gold-standard variants of the popular CoQA task. To execute the transformation code, run model checkpoints, and download both synthetic and gold-standard dialectal benchmark datasets, see http://value-nlp.org.