mlsanityData to 7 Oct 2026

xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection

arXiv 2310.10482 ↗First posted 16 Oct 2023cs.CLcited by 260

Not on the map. Why?

Abstract (arXiv)

Widely used learned metrics for machine translation evaluation, such as COMET and BLEURT, estimate the quality of a translation hypothesis by providing a single sentence-level score. As such, they offer little insight into translation errors (e.g., what are the errors and what is their severity). On the other hand, generative large language models (LLMs) are amplifying the adoption of more granular strategies to evaluation, attempting to detail and categorize translation errors. In this work, we introduce xCOMET, an open-source learned metric designed to bridge the gap between these approaches. xCOMET integrates both sentence-level evaluation and error span detection capabilities, exhibiting state-of-the-art performance across all types of evaluation (sentence-level, system-level, and error span detection). Moreover, it does so while highlighting and categorizing error spans, thus enriching the quality assessment. We also provide a robustness analysis with stress tests, and show that xCOMET is largely capable of identifying localized critical errors and hallucinations.

Key facts. Cited by 260 papers in the map (data through 7 Oct 2026). Builds on 34 of its 66 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

34 of 66 references.

References · 34

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  3. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  6. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  7. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  8. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  9. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  10. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  11. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  12. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
22 more
  1. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  2. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  3. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  4. 2021Learning Compact Metrics for MTcited by 93
  5. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  6. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  7. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  8. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71
  9. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 70
  10. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 60
  11. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  12. 2020MLQE-PE: A Multilingual Quality Estimation and Post-Editing Datasetcited by 47
  13. 2022Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMETcited by 41
  14. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  15. 2023HalOmi: A Manually Annotated Benchmark for Multilingual Hallucination and Omission Detection in Machine Translationcited by 35
  16. 2022UniTE: Unified Translation Evaluationcited by 28
  17. 2022IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languagescited by 27
  18. 2022SALTED: A Framework for SAlient Long-Tail Translation Error Detectioncited by 23
  19. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15
  20. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  21. 2023Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Levelcited by 9
  22. 2023Towards Fine-Grained Information: Identifying the Type and Location of Translation Errorscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 64; 2025: 98; 2026: 96.

260 papers.

Sort:

By area:

  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  2. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  3. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 92
  4. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 52
  5. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  6. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  7. 2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learningcited by 48
  8. 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40
  9. 2023LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedbackcited by 39
  10. 2025Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Studycited by 38
  11. 2024TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinementcited by 37
  12. 2024X-ALMA: Plug & Play Modules and Adaptive Rejection for Quality Translation at Scalecited by 36
38 more
  1. 2024mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generationcited by 32
  2. 2024Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracycited by 25
  3. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  4. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  5. 2025BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Modelscited by 23
  6. 2024Guiding Large Language Models to Post-Edit Machine Translation with Error Annotationscited by 22
  7. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  8. 2024Linear-time Minimum Bayes Risk Decoding with Reference Aggregationcited by 21
  9. 2024Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!cited by 21
  10. 2025R1-T1: Fully Incentivizing Translation Capability in LLMs via Reasoning Learningcited by 21
  11. 2025Seed-X: Building Strong Multilingual Translation LLM with 7B Parameterscited by 21
  12. 2024AI-Assisted Human Evaluation of Machine Translationcited by 20
  13. 2024Machine Translation Meta Evaluation through Translation Accuracy Challenge Setscited by 18
  14. 2024HyperCLOVA X Technical Reportcited by 18
  15. 2024xTower: A Multilingual LLM for Explaining and Correcting Translation Errorscited by 17
  16. 2024Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translationcited by 16
  17. 2024Can Automatic Metrics Assess High-Quality Translations?cited by 16
  18. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  19. 2024PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluationcited by 15
  20. 2024MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferencescited by 15
  21. 2024Mitigating Metric Bias in Minimum Bayes Risk Decodingcited by 15
  22. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
  23. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 14
  24. 2024Improving Minimum Bayes Risk Decoding with Multi-Promptcited by 13
  25. 2024Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Modelscited by 13
  26. 2024mbrs: A Library for Minimum Bayes Risk Decodingcited by 13
  27. 2024Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?cited by 13
  28. 2024M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluationcited by 13
  29. 2025Compositional Translation: A Novel LLM-based Approach for Low-resource Machine Translationcited by 13
  30. 2025SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translationcited by 13
  31. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
  32. 2025Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Modelscited by 11
  33. 2024Pitfalls and Outlooks in Using COMETcited by 10
  34. 2024Beyond Correlation: Interpretable Evaluation of Machine Translation Metricscited by 10
  35. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  36. 2024SpeechQE: Estimating the Quality of Direct Speech Translationcited by 10
  37. 2024Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varietiescited by 10
  38. 2025SwiLTra-Bench: The Swiss Legal Translation Benchmarkcited by 10

50 most cited of 260 All 260 (JSON)