mlsanityData to 7 Oct 2026

MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Task

arXiv 2510.24707 ↗First posted 28 Oct 2025cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

In this paper, we present our submissions to the unified WMT25 Translation Evaluation Shared Task. For the Quality Score Prediction subtask, we create a new generation of MetricX with improvements in the input format and the training protocol, while for the Error Span Detection subtask we develop a new model, GemSpanEval, trained to predict error spans along with their severities and categories. Both systems are based on the state-of-the-art multilingual open-weights model Gemma 3, fine-tuned on publicly available WMT data. We demonstrate that MetricX-25, adapting Gemma 3 to an encoder-only architecture with a regression head on top, can be trained to effectively predict both MQM and ESA quality scores, and significantly outperforms its predecessor. Our decoder-only GemSpanEval model, on the other hand, we show to be competitive in error span detection with xCOMET, a strong encoder-only sequence-tagging baseline. With error span detection formulated as a generative task, we instruct the model to also output the context for each predicted error span, thus ensuring that error spans are identified unambiguously.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

22 of 43 references.

References · 22

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  6. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  7. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  8. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  9. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  10. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
  11. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  12. 2021Learning Compact Metrics for MTcited by 93
10 more
  1. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 92
  2. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  3. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71
  4. 2024Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translationcited by 62
  5. 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40
  6. 2024Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracycited by 25
  7. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  8. 2024PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluationcited by 15
  9. 2025Adapting Decoder-Based Language Models for Diverse Encoder Downstream Taskscited by 10
  10. 2024From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Setcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluationcited by 3
  2. 2026OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agentscited by 2
  3. 2026PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translationcited by 1
  4. 2026What Are We Measuring in NLG? A Meta-Analysis of Evaluation Trends 2020-2025cited by 0
  5. 2026LuxMT Technical Reportcited by 0
  6. 2026Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translationcited by 0
  7. 2026Span-Level Machine Translation Meta-Evaluationcited by 0
  8. 2026CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMscited by 0
  9. 2026Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarioscited by 0
  10. 2026Dynamically Allocating Evaluation Effort for Model Rankingcited by 0