mlsanityData to 7 Oct 2026

COMET: A Neural Framework for MT Evaluation

arXiv 2009.09025 ↗First posted 18 Sep 2020cs.CLcited by 1,024

Not on the map. Why?

Cited by 34 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present COMET, a neural framework for training multilingual machine translation evaluation models which obtains new state-of-the-art levels of correlation with human judgements. Our framework leverages recent breakthroughs in cross-lingual pretrained language modeling resulting in highly multilingual and adaptable MT evaluation models that exploit information from both the source input and a target-language reference translation in order to more accurately predict MT quality. To showcase our framework, we train three models with different types of human judgements: Direct Assessments, Human-mediated Translation Edit Rate and Multidimensional Quality Metrics. Our models achieve new state-of-the-art performance on the WMT 2019 Metrics shared task and demonstrate robustness to high-performing systems.

Key facts. Cited by 1,024 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 130; 2023: 186; 2024: 237; 2025: 235; 2026: 198.

1,024 papers.

Sort:

By area:

  1. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  2. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  3. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  4. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  5. 2023GPTScore: Evaluate as You Desirecited by 357
  6. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  7. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  8. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  9. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  10. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  11. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  12. 2022In-context Examples Selection for Machine Translationcited by 172
38 more
  1. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  2. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  3. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  4. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  5. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  6. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  7. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
  8. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  9. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  10. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  11. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 108
  12. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  13. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  14. 2021Learning Compact Metrics for MTcited by 93
  15. 2021Survey of Low-Resource Machine Translationcited by 92
  16. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 92
  17. 2023Document-Level Machine Translation with Large Language Modelscited by 90
  18. 2022On Decoding Strategies for Neural Text Generatorscited by 89
  19. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  20. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  21. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
  22. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  23. 2022Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Bettercited by 75
  24. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  25. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71
  26. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  27. 2023Extrapolating Large Language Models to Non-English by Aligning Languagescited by 69
  28. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 69
  29. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  30. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 66
  31. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  32. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  33. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 60
  34. 2024Adapting Large Language Models for Document-Level Machine Translationcited by 59
  35. 2022Contrastive Search Is What You Need For Neural Text Generationcited by 58
  36. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 58
  37. 2023Tokenization and the Noiseless Channelcited by 58
  38. 2023Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translationcited by 57

50 most cited of 1,024 All 1,024 (JSON)