mlsanityData to 7 Oct 2026

Trained MT Metrics Learn to Cope with Machine-translated References

arXiv 2312.00536 ↗First posted 1 Dec 2023cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Neural metrics trained on human evaluations of MT tend to correlate well with human judgments, but their behavior is not fully understood. In this paper, we perform a controlled experiment and compare a baseline metric that has not been trained on human evaluations (Prism) to a trained version of the same metric (Prism+FT). Surprisingly, we find that Prism+FT becomes more robust to machine-translated references, which are a notorious problem in MT evaluation. This suggests that the effects of metric training go beyond the intended effect of improving overall correlation with human judgments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 39 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

17 of 39 references.

References · 17

  1. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  6. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  7. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  8. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  9. 2020BLEU might be Guilty but References are not Innocentcited by 109
  10. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  11. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  12. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
5 more
  1. 2019Post-editese: an Exacerbated Translationesecited by 29
  2. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15
  3. 2019Pairwise Neural Machine Translation Evaluationcited by 11
  4. 2021Contrastive Learning for Context-aware Neural Machine TranslationUsing Coreference Informationcited by 9
  5. 2019Automatic Quality Estimation for Natural Language Generation: Ranting (Jointly Rating and Ranking)cited by 4

Built on by

Not cited yet.