# A Benchmark for Evaluating Machine Translation Metrics on Dialects Without Standard Orthography

arXiv: https://arxiv.org/abs/2311.16865 · HTML: https://mlsanity.com/paper/2311.16865/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2023.

## Built on by (most cited first, 11 of 11)

- [Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art](https://mlsanity.com/paper/2406.03930/) (2024)
- [What Do Dialect Speakers Want? A Survey of Attitudes Towards Language Technology for German Dialects](https://mlsanity.com/paper/2402.11968/) (2024)
- [High-Fidelity Simultaneous Speech-To-Speech Translation](https://mlsanity.com/paper/2502.03382/) (2025)
- [A Multi-Dialectal Dataset for German Dialect ASR and Dialect-to-Standard Speech Translation](https://mlsanity.com/paper/2506.02894/) (2025)
- [Simultaneous Speech-to-Speech Translation Without Aligned Data](https://mlsanity.com/paper/2602.11072/) (2026)
- [Make Every Letter Count: Building Dialect Variation Dictionaries from Monolingual Corpora](https://mlsanity.com/paper/2509.17855/) (2025)
- [Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect](https://mlsanity.com/paper/2401.14400/) (2024)
- [Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translation](https://mlsanity.com/paper/2511.06680/) (2025)
- [Vuyko Mistral: Adapting LLMs for Low-Resource Dialectal Translation](https://mlsanity.com/paper/2506.07617/) (2025)
- [FUSE : A Ridge and Random Forest-Based Metric for Evaluating MT in Indigenous Languages](https://mlsanity.com/paper/2504.00021/) (2025)
- [When the Gold Standard Isn't Necessarily Standard: Challenges of Evaluating the Translation of User-Generated Content](https://mlsanity.com/paper/2512.17738/) (2025)

## Abstract (arXiv)

For sensible progress in natural language processing, it is important that we are aware of the limitations of the evaluation metrics we use. In this work, we evaluate how robust metrics are to non-standardized dialects, i.e. spelling differences in language varieties that do not have a standard orthography. To investigate this, we collect a dataset of human translations and human judgments for automatic machine translations from English to two Swiss German dialects. We further create a challenge set for dialect variation and benchmark existing metrics' performances. Our results show that existing metrics cannot reliably evaluate Swiss German text generation outputs, especially on segment level. We propose initial design adaptations that increase robustness in the face of non-standardized dialects, although there remains much room for further improvement. The dataset, code, and models are available here: https://github.com/textshuttle/dialect_eval