mlsanityData to 14 Oct 2026

Multi-VALUE: A Framework for Cross-Dialectal English NLP

arXiv 2212.08011 ↗First posted 15 Dec 2022cs.CLcited by 53

Not on the map. Why?

Abstract (arXiv)

Dialect differences caused by regional, social, and economic factors cause performance discrepancies for many groups of language technology users. Inclusive and equitable language technology must critically be dialect invariant, meaning that performance remains constant over dialectal shifts. Current systems often fall short of this ideal since they are designed and tested on a single dialect: Standard American English (SAE). We introduce a suite of resources for evaluating and achieving English dialect invariance. The resource is called Multi-VALUE, a controllable rule-based translation system spanning 50 English dialects and 189 unique linguistic features. Multi-VALUE maps SAE to synthetic forms of each dialect. First, we use this system to stress tests question answering, machine translation, and semantic parsing. Stress tests reveal significant performance disparities for leading models on non-standard dialects. Second, we use this system as a data augmentation technique to improve the dialect robustness of existing systems. Finally, we partner with native speakers of Chicano and Indian English to release new gold-standard variants of the popular CoQA task. To execute the transformation code, run model checkpoints, and download both synthetic and gold-standard dialectal benchmark datasets, see http://value-nlp.org.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Builds on 40 of its 104 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

40 of 104 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  7. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  9. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  10. 2018CoQA: A Conversational Question Answering Challengecited by 683
  11. 2019How multilingual is Multilingual BERT?cited by 645
  12. 2017Six Challenges for Neural Machine Translationcited by 335
28 more
  1. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  2. 2018Characterizing and Avoiding Negative Transfercited by 229
  3. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 209
  4. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 196
  5. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 183
  6. 2019Choosing Transfer Languages for Cross-Lingual Learningcited by 144
  7. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
  8. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 96
  9. 2020Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT modelcited by 87
  10. 2017Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American Englishcited by 71
  11. 2019Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootingscited by 67
  12. 2022Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selectioncited by 53
  13. 2022Perturbation Augmentation for Fairer NLPcited by 52
  14. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 47
  15. 2022VALUE: Understanding Dialect Disparity in NLUcited by 47
  16. 2021SD-QA: Spoken Dialectal Question Answering for the Real Worldcited by 38
  17. 2017Natural Language Inference over Interaction Spacecited by 34
  18. 2018Anaphora and Coreference Resolution: A Reviewcited by 31
  19. 2020Learning to Recognize Dialect Featurescited by 29
  20. 2020From SPMRL to NMRL: What Did We Learn (and Unlearn) in a Decade of Parsing Morphologically-Rich Languages (MRLs)?cited by 26
  21. 2019Technical report on Conversational Question Answeringcited by 19
  22. 2021Mitigating Racial Biases in Toxic Language Detection with an Equity-Based Ensemble Frameworkcited by 19
  23. 2018Can LSTM Learn to Capture Agreement? The Case of Basquecited by 17
  24. 2022Dialect-robust Evaluation of Generated Textcited by 14
  25. 2021The Possible, the Plausible, and the Desirable: Event-Based Modality Detection for Language Processingcited by 13
  26. 2021To Protect and To Serve? Analyzing Entity-Centric Framing of Police Violencecited by 12
  27. 2019Improving Sentiment Analysis with Multi-task Learning of Negationcited by 8
  28. 2017The Effect of Negators, Modals, and Degree Adverbs on Sentiment Compositioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 14; 2025: 15; 2026: 17.

53 papers.

Sort:

By area:

  1. 2023Can Large Language Models Transform Computational Social Science?cited by 250
  2. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  3. 2024Linguistic Bias in ChatGPT: Language Models Reinforce Dialect Discriminationcited by 56
  4. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 50
  5. 2024Natural Language Processing for Dialects of a Language: A Surveycited by 40
  6. 2025Culture is Not Trivia: Sociocultural Theory for Cultural NLPcited by 40
  7. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  8. 2023A Material Lens on Coloniality in NLPcited by 21
  9. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
  10. 2023A Survey of Corpora for Germanic Low-Resource Languages and Dialectscited by 16
  11. 2023TADA: Task-Agnostic Dialect Adapters for Englishcited by 15
  12. 2023Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classificationscited by 15
38 more
  1. 2024Evaluating Dialect Robustness of Language Models via Conversation Understandingcited by 14
  2. 2023DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rulescited by 13
  3. 2025Tokenization is Sensitive to Language Variationcited by 13
  4. 2024Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varietiescited by 12
  5. 2024Voices Unheard: NLP Resources and Models for Yor`ub'a Regional Dialectscited by 11
  6. 2025Out of Style: RAG's Fragility to Linguistic Variationcited by 11
  7. 2024Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Modelscited by 6
  8. 2024Language and Speech Technology for Central Kurdish Varietiescited by 5
  9. 2024Native Design Bias: Studying the Impact of English Nativeness on Language Model Performancecited by 5
  10. 2024BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of Englishcited by 5
  11. 2025Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarkscited by 5
  12. 2024Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varietiescited by 4
  13. 2025Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varietiescited by 4
  14. 2025When Claims Evolve: Evaluating and Enhancing the Robustness of Embedding Models Against Misinformation Editscited by 3
  15. 2025DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generationcited by 3
  16. 2026How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipelinecited by 3
  17. 2025Analysis of LLM as a grammatical feature tagger for African American Englishcited by 2
  18. 2025A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systemscited by 2
  19. 2025Culture Cartography: Mapping the Landscape of Cultural Knowledgecited by 2
  20. 2024Dialetto, ma Quanto Dialetto? Transcribing and Evaluating Dialects on a Continuumcited by 1
  21. 2025Finding A Voice: Exploring the Potential of African American Dialect and Voice Generation for Chatbotscited by 1
  22. 2025DialUp! Modeling the Language Continuum by Adapting Models to Dialects and Dialects to Modelscited by 1
  23. 2025Harnessing Test-time Adaptation for NLU tasks Involving Dialects of Englishcited by 1
  24. 2026MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languagescited by 1
  25. 2026DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialectscited by 1
  26. 2026DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptationcited by 1
  27. 2026Register Bias in Complexity-Based Large Language Model Routingcited by 1
  28. 2025Nek Minit: Harnessing Pragmatic Metacognitive Prompting for Explainable Sarcasm Detection of Australian and Indian Englishcited by 0
  29. 2025Social Perceptions of English Spelling Variation on Twitter: A Comparative Analysis of Human and LLM Responsescited by 0
  30. 2026DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American Englishcited by 0
  31. 2026A Catalog of Basque Dialectal Resources: Online Collections and Standard-to-Dialectal Adaptationscited by 0
  32. 2026Resource-Lean Lexicon Induction for German Dialectscited by 0
  33. 2026An Investigation of Linguistic Biases in LLM-Based Recommendationscited by 0
  34. 2026LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steeringcited by 0
  35. 2026Who Pays More for Safety? Measuring the Disparate Cost of Safety Alignment across Languagescited by 0
  36. 2026The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipelinecited by 0
  37. 2026Not Safe for All: Auditing the Dialect Penalty in Text-to-Image Safety Pipelinescited by 0
  38. 2026TeochewBench: A Human-Reviewed Benchmark for Teochew Hanzi Translationcited by 0

50 most cited of 53 All 53 (JSON)