mlsanityData to 7 Oct 2026

WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects

arXiv 2502.12404 ↗First posted 18 Feb 2025cs.CLcited by 79

Not on the map. Why?

Abstract (arXiv)

As large language models (LLM) become more and more capable in languages other than English, it is important to collect benchmark datasets in order to evaluate their multilingual performance, including on tasks like machine translation (MT). In this work, we extend the WMT24 dataset to cover 55 languages by collecting new human-written references and post-edits for 46 new languages and dialects in addition to post-edits of the references in 8 out of 9 languages in the original WMT24 dataset. The dataset covers four domains: literary, news, social, and speech. We benchmark a variety of MT providers and LLMs on the collected dataset using automatic metrics and find that LLMs are the best-performing MT systems in all 55 languages. These results should be confirmed using a human-based evaluation, which we leave for future work.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Feb 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 35; 2026: 44.

79 papers.

Sort:

By area:

  1. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  2. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluationcited by 73
  3. 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40
  4. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 26
  5. 2025T5Gemma 2: Seeing, Reading, and Understanding Longercited by 12
  6. 2025Multilingual Contextualization of Large Language Models for Document-Level Machine Translationcited by 11
  7. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  8. 2025Please Translate Again: Two Simple Experiments on Whether Human-Like Reasoning Helps Translationcited by 7
  9. 2025DocHPLT: A Massively Multilingual Document-Level Translation Datasetcited by 7
  10. 2026TranslateGemma Technical Reportcited by 7
  11. 2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Taskscited by 6
  12. 2025HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Modelscited by 6
38 more
  1. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 6
  2. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  3. 2025TACTIC: Translation Agents with Cognitive-Theoretic Interactive Collaborationcited by 5
  4. 2025The Art of Asking: Multilingual Prompt Optimization for Synthetic Datacited by 5
  5. 2025What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representationscited by 4
  6. 2025Expanding the WMT24++ Benchmark with Rumantsch Grischun, Sursilvan, Sutsilvan, Surmiran, Puter, and Valladercited by 4
  7. 2025NiuTrans.LMT: Toward Inclusive and Scalable Multilingual Machine Translation with LLMscited by 4
  8. 2026Scaling Model and Data for Multilingual Machine Translation with Open Large Language Modelscited by 4
  9. 2025From SALAMANDRA to SALAMANDRATA: BSC Submission for WMT25 General Machine Translation Shared Taskcited by 3
  10. 2026NeoAMT: Neologism-Aware Agentic Machine Translation with Reinforcement Learningcited by 3
  11. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Expertscited by 3
  12. 2025Mutarjim: Advancing Bidirectional Arabic-English Translation with a Small Language Modelcited by 2
  13. 2025Treasure Hunt: Real-time Targeting of the Long Tail using Training-Time Markerscited by 2
  14. 2025VocabTailor: Dynamic Vocabulary Selection for Downstream Tasks in Small Language Modelscited by 2
  15. 2025Penalizing Length: Uncovering Systematic Bias in Quality Estimation Metricscited by 2
  16. 2026K-EXAONE Technical Reportcited by 2
  17. 2026GRRM: Group Relative Reward Modeling for Machine Translationcited by 2
  18. 2026RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deploymentcited by 2
  19. 2025The Effect of Language Diversity When Fine-Tuning Large Language Models for Translationcited by 1
  20. 2025TransBench: Benchmarking Machine Translation for Industrial-Scale Applicationscited by 1
  21. 2025OpenWHO: A Document-Level Parallel Corpus for Health Translation in Low-Resource Languagescited by 1
  22. 2025ACADATA: Parallel Dataset of Academic Data for Machine Translationcited by 1
  23. 2025Steering LLMs toward Korean Local Speech: Iterative Refinement Framework for Faithful Dialect Translationcited by 1
  24. 2025Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotationscited by 1
  25. 2026Alexandria: A Multi-Domain Dialectal Arabic Machine Translation Dataset for Culturally Inclusive and Linguistically Diverse LLMscited by 1
  26. 2026PEGRL: Improving Machine Translation by Post-Editing Guided Reinforcement Learningcited by 1
  27. 2026Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasetscited by 1
  28. 2026TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representationcited by 1
  29. 2026Gender Disambiguation in Machine Translation: Diagnostic Evaluation in Decoder-Only Architecturescited by 1
  30. 2026Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varietiescited by 1
  31. 2026Optimizing Korean-Centric LLMs via Token Pruningcited by 1
  32. 2026Incentivizing Parametric Knowledge via Reinforcement Learning with Verifiable Rewards for Cross-Cultural Entity Translationcited by 1
  33. 2026Beyond "To whom it may concern": Tailoring Machine Translation to Audience and Intentcited by 1
  34. 2026ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairscited by 1
  35. 2026Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustnesscited by 1
  36. 2026Last Translation Benchmarkcited by 1
  37. 2026Improving Term Evaluation in Machine Translation: Variation Matterscited by 1
  38. 2025Discriminating Form and Meaning in Multilingual Models with Minimal-Pair ABX Taskscited by 0

50 most cited of 79 All 79 (JSON)