# Preliminary WMT24 Ranking of General MT Systems and LLMs

arXiv: https://arxiv.org/abs/2407.19884 · HTML: https://mlsanity.com/paper/2407.19884/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 11 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Builds on (most cited first, 3 of 3)

- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies](https://mlsanity.com/paper/2401.06760/) (2024)
- [Machine Translation Meta Evaluation through Translation Accuracy Challenge Sets](https://mlsanity.com/paper/2401.16313/) (2024)

## Built on by (most cited first, 20 of 20)

- [EuroLLM: Multilingual Language Models for Europe](https://mlsanity.com/paper/2409.16235/) (2024)
- [MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learning](https://mlsanity.com/paper/2504.10160/) (2025)
- [Translating Step-by-Step: Decomposing the Translation Process for Improved Translation Quality of Long-Form Texts](https://mlsanity.com/paper/2409.06790/) (2024)
- [How to Select Datapoints for Efficient Human Evaluation of NLG Models?](https://mlsanity.com/paper/2501.18251/) (2025)
- [R1-T1: Fully Incentivizing Translation Capability in LLMs via Reasoning Learning](https://mlsanity.com/paper/2502.19735/) (2025)
- [Multilingual Language Model Pretraining using Machine-translated Data](https://mlsanity.com/paper/2502.13252/) (2025)
- [Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation](https://mlsanity.com/paper/2408.12780/) (2024)
- [On the Implications of Verbose LLM Outputs: A Case Study in Translation Evaluation](https://mlsanity.com/paper/2410.00863/) (2024)
- [Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Data](https://mlsanity.com/paper/2408.06537/) (2024)
- [Pitfalls and Outlooks in Using COMET](https://mlsanity.com/paper/2408.15366/) (2024)
- [Towards Automatic Evaluation for Image Transcreation](https://mlsanity.com/paper/2412.13717/) (2024)
- [Multilingual Pretraining Using a Large Corpus Machine-Translated from a Single Source Language](https://mlsanity.com/paper/2410.23956/) (2024)
- [ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilities](https://mlsanity.com/paper/2506.12376/) (2025)
- [IKUN for WMT24 General MT Task: LLMs Are here for Multilingual Machine Translation](https://mlsanity.com/paper/2408.11512/) (2024)
- [A Context-aware Framework for Translation-mediated Conversations](https://mlsanity.com/paper/2412.04205/) (2024)
- [Dialectal and Low-Resource Machine Translation for Aromanian](https://mlsanity.com/paper/2410.17728/) (2024)
- [MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insights](https://mlsanity.com/paper/2606.07020/) (2026)
- [Plug, Play, and Fuse: Zero-Shot Joint Decoding via Word-Level Re-ranking Across Diverse Vocabularies](https://mlsanity.com/paper/2408.11327/) (2024)
- [Cogs in a Machine, Doing What They're Meant to Do -- The AMI Submission to the WMT24 General Translation Task](https://mlsanity.com/paper/2410.03381/) (2024)
- [Human + AI for Accelerating Ad Localization Evaluation](https://mlsanity.com/paper/2509.12543/) (2025)

## Abstract (arXiv)

This is the preliminary ranking of WMT24 General MT systems based on automatic metrics. The official ranking will be a human evaluation, which is superior to the automatic ranking and supersedes it. The purpose of this report is not to interpret any findings but only provide preliminary results to the participants of the General MT task that may be useful during the writing of the system submission.