# Dialect-robust Evaluation of Generated Text arXiv: https://arxiv.org/abs/2211.00922 ยท HTML: https://mlsanity.com/paper/2211.00922/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Builds on 24 of its 55 references that are in the map. - First posted on arXiv 2 Nov 2022. ## Builds on (most cited first, 24 of 24) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020) - [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022) - [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020) - [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020) - [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021) - [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020) - [Demographic Dialectal Variation in Social Media: A Case Study of African-American English](https://mlsanity.com/paper/1608.08868/) (2016) - [Scaling Up Models and Data with textttt5x and textttseqio](https://mlsanity.com/paper/2203.17189/) (2022) - [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022) - [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://mlsanity.com/paper/2004.14564/) (2020) - [Rethinking embedding coupling in pre-trained language models](https://mlsanity.com/paper/2010.12821/) (2020) - [Learning Compact Metrics for MT](https://mlsanity.com/paper/2110.06341/) (2021) - [VALUE: Understanding Dialect Disparity in NLU](https://mlsanity.com/paper/2204.03031/) (2022) - [SD-QA: Spoken Dialectal Question Answering for the Real World](https://mlsanity.com/paper/2109.12072/) (2021) - [Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets](https://mlsanity.com/paper/1709.09500/) (2017) - [Learning to Recognize Dialect Features](https://mlsanity.com/paper/2010.12707/) (2020) - [FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation](https://mlsanity.com/paper/2210.00193/) (2022) - [A Fast, Compact, Accurate Model for Language Identification of Codemixed Text](https://mlsanity.com/paper/1810.04142/) (2018) - [Finding Variants for Construction-Based Dialectometry: A Corpus-Based Approach to Regional CxGs](https://mlsanity.com/paper/2104.01299/) (2021) - [Natural Language Processing with Small Feed-Forward Networks](https://mlsanity.com/paper/1708.00214/) (2017) - [Corpus-Guided Contrast Sets for Morphosyntactic Feature Detection in Low-Resource English Varieties](https://mlsanity.com/paper/2209.07611/) (2022) ## Built on by (most cited first, 14 of 14) - [Multi-VALUE: A Framework for Cross-Dialectal English NLP](https://mlsanity.com/paper/2212.08011/) (2022) - [On the Blind Spots of Model-Based Evaluation Metrics for Text Generation](https://mlsanity.com/paper/2212.10020/) (2022) - [Natural Language Processing for Dialects of a Language: A Survey](https://mlsanity.com/paper/2401.05632/) (2024) - [Exploiting Biased Models to De-bias Text: A Gender-Fair Rewriting Model](https://mlsanity.com/paper/2305.11140/) (2023) - [Evaluating Dialect Robustness of Language Models via Conversation Understanding](https://mlsanity.com/paper/2405.05688/) (2024) - [DADA: Dialect Adaptation via Dynamic Aggregation of Linguistic Rules](https://mlsanity.com/paper/2305.13406/) (2023) - [How Well Do LLMs Identify Cultural Unity in Diversity?](https://mlsanity.com/paper/2408.05102/) (2024) - [Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models](https://mlsanity.com/paper/2409.00358/) (2024) - [BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for Varieties of English](https://mlsanity.com/paper/2412.04726/) (2024) - [Reward Models are Metrics in a Trench Coat](https://mlsanity.com/paper/2510.03231/) (2025) - [Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translation](https://mlsanity.com/paper/2511.06680/) (2025) - [A User-Centered Evaluation of Spanish Text Simplification](https://mlsanity.com/paper/2308.07556/) (2023) - [Variation is the Norm: Embracing Sociolinguistics in NLP](https://mlsanity.com/paper/2603.24222/) (2026) - [On the Role of Citations in Preference Data](https://mlsanity.com/paper/2608.21376/) (2026) ## Abstract (arXiv) Evaluation metrics that are not robust to dialect variation make it impossible to tell how well systems perform for many groups of users, and can even penalize systems for producing text in lower-resource dialects. However, currently, there exists no way to quantify how metrics respond to change in the dialect of a generated utterance. We thus formalize dialect robustness and dialect awareness as goals for NLG evaluation metrics. We introduce a suite of methods and corresponding statistical tests one can use to assess metrics in light of the two goals. Applying the suite to current state-of-the-art metrics, we demonstrate that they are not dialect-robust and that semantic perturbations frequently lead to smaller decreases in a metric than the introduction of dialect features. As a first step to overcome this limitation, we propose a training schema, NANO, which introduces regional and language information to the pretraining process of a metric. We demonstrate that NANO provides a size-efficient way for models to improve the dialect robustness while simultaneously improving their performance on the standard metric benchmark.