mlsanityData to 14 Oct 2026

Expanding the WMT24++ Benchmark with Rumantsch Grischun, Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader

arXiv 2509.03148 ↗First posted 3 Sep 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

The Romansh language, spoken in Switzerland, has limited resources for machine translation evaluation. In this paper, we present a benchmark for six varieties of Romansh: Rumantsch Grischun, a supra-regional variety, and five regional varieties: Sursilvan, Sutsilvan, Surmiran, Puter, and Vallader. Our reference translations were created by human translators based on the WMT24++ benchmark, which ensures parallelism with more than 55 other languages. An automatic evaluation of existing MT systems and LLMs shows that translation out of Romansh into German is handled relatively well for all the varieties, but translation into Romansh is still challenging.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 10 of its 35 references that are in the map. First posted on arXiv 3 Sep 2025.

Builds on

10 of 35 references.

References · 10

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  3. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  4. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  5. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  6. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  7. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  8. 2025WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialectscited by 79
  9. 2019Domain Robustness in Neural Machine Translationcited by 65
  10. 2025SwiLTra-Bench: The Swiss Legal Translation Benchmarkcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Robust Language Identification for Romansh Varietiescited by 2
  2. 2026Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varietiescited by 2
  3. 2026ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairscited by 1
  4. 2026Reinforcement Learning Elicits Contextual Learning of Unseen Language Translationcited by 0