mlsanityData to 14 Oct 2026

MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task

arXiv 2410.03983 ↗First posted 4 Oct 2024cs.CLcited by 93

Not on the map. Why?

Abstract (arXiv)

In this paper, we present the MetricX-24 submissions to the WMT24 Metrics Shared Task and provide details on the improvements we made over the previous version of MetricX. Our primary submission is a hybrid reference-based/-free metric, which can score a translation irrespective of whether it is given the source segment, the reference, or both. The metric is trained on previous WMT data in a two-stage fashion, first on the DA ratings only, then on a mixture of MQM and DA ratings. The training set in both stages is augmented with synthetic examples that we created to make the metric more robust to several common failure modes, such as fluent but unrelated translation, or undertranslation. We demonstrate the benefits of the individual modifications via an ablation study, and show a significant performance increase over MetricX-23 on the WMT23 MQM ratings, as well as our new synthetic challenge set.

Key facts. Cited by 93 papers in the map (data through 14 Oct 2026). Builds on 19 of its 34 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

19 of 34 references.

References · 19

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  6. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  7. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  8. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  9. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 207
  10. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  11. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  12. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
7 more
  1. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  2. 2021Learning Compact Metrics for MTcited by 93
  3. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 74
  4. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  5. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  6. 2024PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluationcited by 16
  7. 2022Alibaba-Translate China's Submission for WMT 2022 Metrics Shared Taskcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 44; 2026: 46.

93 papers.

Sort:

By area:

  1. 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40
  2. 2025Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistencycited by 30
  3. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  4. 2025BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translationcited by 22
  5. 2025SEA-HELM: Southeast Asian Holistic Evaluation of Language Modelscited by 19
  6. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
  7. 2025Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progresscited by 10
  8. 2025MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Taskcited by 10
  9. 2025AskQE: Question Answering as Automatic Evaluation for Machine Translationcited by 7
  10. 2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Taskscited by 7
  11. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 7
  12. 2026TranslateGemma Technical Reportcited by 7
38 more
  1. 2025Extending Automatic Machine Translation Evaluation to Book-Length Documentscited by 6
  2. 2026Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluationcited by 6
  3. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  4. 2025Batayan: A Filipino NLP benchmark for evaluating Large Language Modelscited by 5
  5. 2025UNCERTAINTY-LINE: Length-Invariant Estimation of Uncertainty for Large Language Modelscited by 5
  6. 2025TACTIC: Translation Agents with Cognitive-Theoretic Interactive Collaborationcited by 5
  7. 2025Translate Smart, not Hard: Cascaded Translation Systems with Quality-Aware Deferralcited by 4
  8. 2025You Cannot Feed Two Birds with One Score: the Accuracy-Naturalness Tradeoff in Translationcited by 4
  9. 2025KIT's Offline Speech Translation and Instruction Following Submission for IWSLT 2025cited by 4
  10. 2025Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translationcited by 4
  11. 2026Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translationcited by 4
  12. 2024Domain-Specific Translation with Open-Source Large Language Models: Resource-Oriented Analysiscited by 3
  13. 2025Theoretical Guarantees for Minimum Bayes Risk Decodingcited by 3
  14. 2025TAT-R1: Terminology-Aware Translation with Reinforcement Learning and Word Alignmentcited by 3
  15. 2025How to Evaluate Speech Translation with Source-Aware Neural MT Metricscited by 3
  16. 2025DiscoX: Benchmarking Discourse-Level Translation task in Expert Domainscited by 3
  17. 2024From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Setcited by 2
  18. 2025Early-Exit and Instant Confidence Translation Quality Estimationcited by 2
  19. 2025Test-Time Alignment for Large Language Models via Textual Model Predictive Controlcited by 2
  20. 2025Calibrating Translation Decoding with Quality Estimation on LLMscited by 2
  21. 2025Evaluating LLMs on Chinese Idiom Translationcited by 2
  22. 2025PLaMo 2 Technical Reportcited by 2
  23. 2025Penalizing Length: Uncovering Systematic Bias in Quality Estimation Metricscited by 2
  24. 2025MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translationcited by 2
  25. 2025Agreement-Constrained Probabilistic Minimum Bayes Risk Decodingcited by 2
  26. 2026Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domainscited by 2
  27. 2026Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translationscited by 2
  28. 2024Multi-perspective Alignment for Increasing Naturalness in Neural Machine Translationcited by 1
  29. 2025Slim attention: cut your context memory in half without loss -- K-cache is all you need for MHAcited by 1
  30. 2025ContrastScore: Towards Higher Quality, Less Biased, More Efficient Evaluation Metrics with Contrastive Evaluationcited by 1
  31. 2025TransBench: Benchmarking Machine Translation for Industrial-Scale Applicationscited by 1
  32. 2025A French Version of the OLDI Seed Corpuscited by 1
  33. 2025TopXGen: Topic-Diverse Parallel Data Generation for Low-Resource Machine Translationcited by 1
  34. 2025Preliminary Ranking of WMT25 General Machine Translation Systemscited by 1
  35. 2025OpenWHO: A Document-Level Parallel Corpus for Health Translation in Low-Resource Languagescited by 1
  36. 2025Evaluating the Impact of Verbal Multiword Expressions on Machine Translationcited by 1
  37. 2025Case-Based Decision-Theoretic Decoding with Quality Memoriescited by 1
  38. 2025Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluationcited by 1

50 most cited of 93 All 93 (JSON)