mlsanityData to 14 Oct 2026

CODET: A Benchmark for Contrastive Dialectal Evaluation of Machine Translation

arXiv 2305.17267 ↗First posted 26 May 2023cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Neural machine translation (NMT) systems exhibit limited robustness in handling source-side linguistic variations. Their performance tends to degrade when faced with even slight deviations in language usage, such as different domains or variations introduced by second-language speakers. It is intuitive to extend this observation to encompass dialectal variations as well, but the work allowing the community to evaluate MT systems on this dimension is limited. To alleviate this issue, we compile and release CODET, a contrastive dialectal benchmark encompassing 891 different variations from twelve different languages. We also quantitatively demonstrate the challenges large MT models face in effectively translating dialectal variants. All the data and code have been released.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

22 of 43 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  8. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  9. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  10. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
  11. 2020TICO-19: the Translation Initiative for Covid-19cited by 68
  12. 2019On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Modelscited by 56
10 more
  1. 2018MTNT: A Testbed for Machine Translation of Noisy Textcited by 54
  2. 2020Not Low-Resource Anymore: Aligner Ensembling, Batch Filtering, and New Datasets for Bengali-English Machine Translationcited by 53
  3. 2021DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoderscited by 50
  4. 2022Using natural language prompts for machine translationcited by 27
  5. 2021Machine Translation into Low-resource Language Varietiescited by 19
  6. 2018Part-of-Speech Tagging on an Endangered Language: a Parallel Griko-Italian Resourcecited by 10
  7. 2017Machine Translation of Low-Resource Spoken Dialects: Strategies for Normalizing Swiss Germancited by 9
  8. 2020Leveraging Multilingual News Websites for Building a Kurdish Parallel Corpuscited by 9
  9. 2019An Analysis of Source-Side Grammatical Errors in NMTcited by 6
  10. 2023Approaches to Corpus Creation for Low-Resource Language Technology: the Case of Southern Kurdish and Lakicited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 2.

9 papers.

  1. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 50
  2. 2024Natural Language Processing for Dialects of a Language: A Surveycited by 40
  3. 2024NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Taskcited by 23
  4. 2024Voices Unheard: NLP Resources and Models for Yor`ub'a Regional Dialectscited by 11
  5. 2025Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challengescited by 6
  6. 2024Language and Speech Technology for Central Kurdish Varietiescited by 5
  7. 2025Lost in Variation? Evaluating NLI Performance in Basque and Spanish Geographical Variantscited by 2
  8. 2026Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustnesscited by 1
  9. 2026A Catalog of Basque Dialectal Resources: Online Collections and Standard-to-Dialectal Adaptationscited by 0