mlsanityData to 7 Oct 2026

Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains

arXiv 2402.18747 ↗First posted 28 Feb 2024cs.CL cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

We introduce a new, extensive multidimensional quality metrics (MQM) annotated dataset covering 11 language pairs in the biomedical domain. We use this dataset to investigate whether machine translation (MT) metrics which are fine-tuned on human-generated MT quality judgements are robust to domain shifts between training and inference. We find that fine-tuned metrics exhibit a substantial performance drop in the unseen domain scenario relative to metrics that rely on the surface form, as well as pre-trained metrics which are not fine-tuned on MT quality judgments.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 18 of its 38 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

18 of 38 references.

References · 18

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  6. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  7. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  8. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  9. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  10. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  11. 2018Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluationcited by 88
  12. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56
6 more
  1. 2022BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generationcited by 48
  2. 2020Paraphrase Generation as Zero-Shot Multilingual Translation: Disentangling Semantic Similarity from Lexical and Syntactic Diversitycited by 35
  3. 2024A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelismcited by 33
  4. 2020Reassessing Claims of Human Parity and Super-Human Performance in Machine Translation at WMT 2019cited by 20
  5. 2023Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Referencecited by 11
  6. 2023Tailoring Domain Adaptation for Machine Translation Quality Estimationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 8; 2026: 5.

21 papers.

Sort:

By area:

  1. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
  2. 2024Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracycited by 25
  3. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  4. 2024AI-Assisted Human Evaluation of Machine Translationcited by 20
  5. 2024Can Automatic Metrics Assess High-Quality Translations?cited by 16
  6. 2024Pitfalls and Outlooks in Using COMETcited by 10
  7. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  8. 2025Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progresscited by 10
  9. 2024Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?cited by 9
  10. 2024Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translationcited by 8
  11. 2025QE4PE: Word-level Quality Estimation for Human Post-Editingcited by 6
  12. 2025Are Generative Models Underconfident? Better Quality Estimation with Boosted Model Probabilitycited by 4
9 more
  1. 2025Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreementcited by 4
  2. 2025COMET-poly: Machine Translation Metric Grounded in Other Candidatescited by 2
  3. 2025Reward Models are Metrics in a Trench Coatcited by 2
  4. 2026Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domainscited by 2
  5. 2026Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translationscited by 1
  6. 2025Learning to Translate Ambiguous Terminology by Preference Optimization on Post-Editscited by 0
  7. 2026Sigmoid Head for Quality Estimation under Language Ambiguitycited by 0
  8. 2026LocQE: Principled Domain Adaptation for Localisation Quality Estimation by Leveraging Post-Editscited by 0
  9. 2026TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluationcited by 0