mlsanityData to 7 Oct 2026

Pitfalls and Outlooks in Using COMET

arXiv 2408.15366 ↗First posted 27 Aug 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

The COMET metric has blazed a trail in the machine translation community, given its strong correlation with human judgements of translation quality. Its success stems from being a modified pre-trained multilingual model finetuned for quality assessment. However, it being a machine learning model also gives rise to a new set of pitfalls that may not be widely known. We investigate these unexpected behaviours from three aspects: 1) technical: obsolete software versions and compute precision; 2) data: empty content, language mismatch, and translationese at test time as well as distribution and domain biases in training; 3) usage and reporting: multi-reference support and model referencing in the literature. All of these problems imply that COMET scores are not comparable between papers or even technical setups and we put forward our perspective on fixing each issue. Furthermore, we release the sacreCOMET package that can generate a signature for the software and model configuration as well as an appropriate citation. The goal of this work is to help the community make more sound use of the COMET metric.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 33 of its 63 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

33 of 63 references.

References · 33

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  6. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  7. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  8. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  9. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  10. 2020BLEU might be Guilty but References are not Innocentcited by 109
  11. 2023The Semantic Scholar Open Data Platformcited by 109
  12. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
21 more
  1. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  2. 2024Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translationcited by 62
  3. 2020Unbabel's Participation in the WMT20 Metrics Shared Taskcited by 56
  4. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
  5. 2021Uncertainty-Aware Machine Translation Evaluationcited by 35
  6. 2023MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methodscited by 33
  7. 2022ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metricscited by 25
  8. 2023BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Trainingcited by 24
  9. 2024Fine-Tuned Machine Translation Metrics Struggle in Unseen Domainscited by 21
  10. 2024AI-Assisted Human Evaluation of Machine Translationcited by 20
  11. 2024Preliminary WMT24 Ranking of General MT Systems and LLMscited by 20
  12. 2023There's no Data Like Better Data: Using QE Metrics for MT Data Filteringcited by 19
  13. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15
  14. 2018Multi-Reference Training with Pseudo-References for Neural Translation and Text Generationcited by 14
  15. 2024Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimationcited by 13
  16. 2023Poor Man's Quality Estimation: Predicting Reference-Based MT Metrics Without the Referencecited by 11
  17. 2023Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Modelcited by 9
  18. 2022Don't Discard Fixed-Window Audio Segmentation in Speech-to-Text Translationcited by 8
  19. 2024Quality and Quantity of Machine Translation References for Automatic Metricscited by 7
  20. 2024Chasing COMET: Leveraging Minimum Bayes Risk Decoding for Self-Improving Machine Translationcited by 6
  21. 2024Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  2. 2025Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMscited by 6
  3. 2025Better Late Than Never: Meta-Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translationcited by 4
  4. 2025How to Evaluate Speech Translation with Source-Aware Neural MT Metricscited by 3
  5. 2025Penalizing Length: Uncovering Systematic Bias in Quality Estimation Metricscited by 2
  6. 2026Last Translation Benchmarkcited by 1
  7. 2025Assessing the Political Fairness of Multilingual LLMs: A Case Study based on a 21-way Multiparallel EuroParl Datasetcited by 0
  8. 2026LLM-Powered Automatic Translation and Urgency in Crisis Scenarioscited by 0
  9. 2026Hindsight Quality Prediction Experiments in Multi-Candidate Human-Post-Edited Machine Translationcited by 0
  10. 2026Do Evaluation Metrics Detect Errors in Classical Chinese to English Translations?cited by 0