mlsanityData to 7 Oct 2026

VALUE: Understanding Dialect Disparity in NLU

arXiv 2204.03031 ↗First posted 6 Apr 2022cs.CLcited by 45

Not on the map. Why?

Abstract (arXiv)

English Natural Language Understanding (NLU) systems have achieved great performances and even outperformed humans on benchmarks like GLUE and SuperGLUE. However, these benchmarks contain only textbook Standard American English (SAE). Other dialects have been largely overlooked in the NLP community. This leads to biased and inequitable NLU systems that serve only a sub-population of speakers. To understand disparities in current models and to facilitate more dialect-competent NLU systems, we introduce the VernAcular Language Understanding Evaluation (VALUE) benchmark, a challenging variant of GLUE that we created with a set of lexical and morphosyntactic transformation rules. In this initial release (V.1), we construct rules for 11 features of African American Vernacular English (AAVE), and we recruit fluent AAVE speakers to validate each feature transformation via linguistic acceptability judgments in a participatory design manner. Experiments show that these new dialectal features can lead to a drop in model performance. To run the transformation code and download both synthetic and gold-standard dialectal GLUE benchmarks, see https://github.com/SALT-NLP/value

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 29 of its 72 references that are in the map. First posted on arXiv 6 Apr 2022.

Builds on

29 of 72 references.

References · 29

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  7. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  8. 2018Neural Network Acceptability Judgmentscited by 952
  9. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  10. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  11. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  12. 2018CoQA: A Conversational Question Answering Challengecited by 674
17 more
  1. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  2. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  3. 2018Generating Natural Language Adversarial Examplescited by 422
  4. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  5. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 304
  6. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  7. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  8. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 194
  9. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 181
  10. 2017Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American Englishcited by 69
  11. 2020It's Morphin' Time! Combating Linguistic Discrimination with Inflectional Perturbationscited by 64
  12. 2017A causal framework for explaining the predictions of black-box sequence-to-sequence modelscited by 58
  13. 2020Robust Encodings: A Framework for Combating Adversarial Typoscited by 58
  14. 2020Investigating African-American Vernacular English in Transformer-Based Text Generationcited by 52
  15. 2015Computational Sociolinguistics: A Surveycited by 35
  16. 2020Learning to Recognize Dialect Featurescited by 29
  17. 2021Mitigating Racial Biases in Toxic Language Detection with an Equity-Based Ensemble Frameworkcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 11; 2024: 11; 2025: 12; 2026: 8.

45 papers.

Sort:

By area:

  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  2. 2022Prompting GPT-3 To Be Reliablecited by 258
  3. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  4. 2022Multi-VALUE: A Framework for Cross-Dialectal English NLPcited by 51
  5. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 49
  6. 2024Natural Language Processing for Dialects of a Language: A Surveycited by 40
  7. 2023Evaluation of African American Language Bias in Natural Language Generationcited by 36
  8. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  9. 2024Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Taskscited by 20
  10. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  11. 2023A Material Lens on Coloniality in NLPcited by 19
  12. 2023TADA: Task-Agnostic Dialect Adapters for Englishcited by 15
33 more
  1. 2022Dialect-robust Evaluation of Generated Textcited by 14
  2. 2024Evaluating Dialect Robustness of Language Models via Conversation Understandingcited by 13
  3. 2023DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rulescited by 12
  4. 2024AAVENUE: Detecting LLM Biases on NLU Tasks in AAVE via a Novel Benchmarkcited by 11
  5. 2023Task-Agnostic Low-Rank Adapters for Unseen English Dialectscited by 9
  6. 2024Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Modelscited by 6
  7. 2024Standard Language Ideology in AI-Generated Languagecited by 5
  8. 2025Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarkscited by 5
  9. 2023MC2: Towards Transparent and Culturally-Aware NLP for Minority Languages in Chinacited by 4
  10. 2024Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varietiescited by 4
  11. 2025EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Modelscited by 4
  12. 2025Trans-EnV: A Framework for Evaluating the Linguistic Robustness of LLMs Against English Varietiescited by 4
  13. 2024The Life Cycle of Large Language Models: A Review of Biases in Educationcited by 3
  14. 2025MEDEQUALQA: Evaluating Biases in LLMs with Counterfactual Reasoningcited by 3
  15. 2025DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generationcited by 3
  16. 2025Investigating Co-Constructive Behavior of Large Language Models in Explanation Dialoguescited by 2
  17. 2025Artificial Impressions: Evaluating Large Language Model Behavior Through the Lens of Trait Impressionscited by 2
  18. 2026How Does "English (US)" Become the Default? Triangulating Structural Bias Towards American English Across the LLM Pipelinecited by 2
  19. 2023READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noisescited by 1
  20. 2024Disentangling Dialect from Social Bias via Multitask Learning to Improve Fairnesscited by 1
  21. 2025Finding A Voice: Exploring the Potential of African American Dialect and Voice Generation for Chatbotscited by 1
  22. 2025Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translationcited by 1
  23. 2026DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialectscited by 1
  24. 2026DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptationcited by 1
  25. 2026Register Bias in Complexity-Based Large Language Model Routingcited by 1
  26. 2023Evaluating Robustness of Dialogue Summarization Models in the Presence of Naturally Occurring Variationscited by 0
  27. 2023Latent Feature-based Data Splits to Improve Generalisation Evaluation: A Hate Speech Detection Case Studycited by 0
  28. 2025Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varietiescited by 0
  29. 2025Splits! Flexible Sociocultural Linguistic Investigation at Scalecited by 0
  30. 2026DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American Englishcited by 0
  31. 2026LLMs Silently Correct African American English: Auditing and Mitigating Dialect Bias via Activation Steeringcited by 0
  32. 2026Not Safe for All: Auditing the Dialect Penalty in Text-to-Image Safety Pipelinescited by 0
  33. 2026FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditingcited by 0