mlsanityData to 7 Oct 2026

CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task

arXiv 2209.06243 ↗First posted 13 Sep 2022cs.CL cs.LGcited by 205

Not on the map. Why?

Abstract (arXiv)

We present the joint contribution of IST and Unbabel to the WMT 2022 Shared Task on Quality Estimation (QE). Our team participated on all three subtasks: (i) Sentence and Word-level Quality Prediction; (ii) Explainable QE; and (iii) Critical Error Detection. For all tasks we build on top of the COMET framework, connecting it with the predictor-estimator architecture of OpenKiwi, and equipping it with a word-level sequence tagger and an explanation extractor. Our results suggest that incorporating references during pretraining improves performance across several language pairs on downstream tasks, and that jointly training with sentence and word-level objectives yields a further boost. Furthermore, combining attention and gradient information proved to be the top strategy for extracting good explanations of sentence-level QE models. Overall, our submissions achieved the best results for all three tasks for almost all language pairs by a considerable margin.

Key facts. Cited by 205 papers in the map (data through 7 Oct 2026). Builds on 20 of its 35 references that are in the map. First posted on arXiv 13 Sep 2022.

Builds on

20 of 35 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  8. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  9. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  10. 2016Rationalizing Neural Predictionscited by 281
  11. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  12. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
8 more
  1. 2019OpenKiwi: An Open Source Framework for Quality Estimationcited by 49
  2. 2020MLQE-PE: A Multilingual Quality Estimation and Post-Editing Datasetcited by 47
  3. 2016Explaining Predictions of Non-Linear Classifiers in NLPcited by 37
  4. 2022UniTE: Unified Translation Evaluationcited by 28
  5. 2021The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Resultscited by 27
  6. 2022An Empirical Study on Explanations in Out-of-Domain Settingscited by 21
  7. 2020TransQuest at WMT2020: Sentence-Level Direct Assessmentcited by 20
  8. 2021SPECTRA: Sparse Structured Text Rationalizationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 29; 2024: 53; 2025: 69; 2026: 52.

205 papers.

Sort:

By area:

  1. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  2. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  3. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  4. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  5. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 92
  6. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  7. 2023Efficient Benchmarking of Language Modelscited by 71
  8. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 69
  9. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 52
  10. 2025MT-R1-Zero: Advancing LLM-based Machine Translation via R1-Zero-like Reinforcement Learningcited by 48
  11. 2023Reducing Privacy Risks in Online Self-Disclosures with Language Modelscited by 47
  12. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
38 more
  1. 2024Question Translation Training for Better Multilingual Reasoningcited by 41
  2. 2023LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedbackcited by 39
  3. 2024TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinementcited by 37
  4. 2024A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelismcited by 33
  5. 2024DRT: Deep Reasoning Translation via Long Chain-of-Thoughtcited by 28
  6. 2023Translate Meanings, Not Just Words: IdiomKB's Role in Optimizing Idiomatic Translation with Language Modelscited by 23
  7. 2024The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilitiescited by 22
  8. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  9. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  10. 2023There's no Data Like Better Data: Using QE Metrics for MT Data Filteringcited by 19
  11. 2024NEO-BENCH: Evaluating Robustness of Large Language Models with Neologismscited by 19
  12. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  13. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  14. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 16
  15. 2023An Empirical Study of Translation Hypothesis Ensembling with Large Language Modelscited by 16
  16. 2024DUAL-REFLECT: Enhancing Large Language Models for Reflective Translation through Dual Learning Feedback Mechanismscited by 16
  17. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15
  18. 2023Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSAcited by 15
  19. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  20. 2024Mitigating Metric Bias in Minimum Bayes Risk Decodingcited by 15
  21. 2024MT-Ranker: Reference-free machine translation evaluation by inter-system rankingcited by 14
  22. 2025Scaling Low-Resource MT via Synthetic Data Generation with LLMscited by 14
  23. 2022Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translationcited by 13
  24. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  25. 2024Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solutioncited by 13
  26. 2024Improving Minimum Bayes Risk Decoding with Multi-Promptcited by 13
  27. 2024M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluationcited by 13
  28. 2025Krikri: Advancing Open Large Language Models for Greekcited by 13
  29. 2025Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecturecited by 12
  30. 2023SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Windowcited by 11
  31. 2024Ladder: A Model-Agnostic Framework Boosting LLM-based Machine Translation to the Next Levelcited by 11
  32. 2025Lost in Literalism: How Supervised Training Shapes Translationese in LLMscited by 11
  33. 2025ExTrans: Multilingual Deep Reasoning Translation via Exemplar-Enhanced Reinforcement Learningcited by 11
  34. 2024From Handcrafted Features to LLMs: A Brief Survey for Machine Translation Quality Estimationcited by 10
  35. 2024Creative and Context-Aware Translation of East Asian Idioms with GPT-4cited by 10
  36. 2024Beyond Correlation: Interpretable Evaluation of Machine Translation Metricscited by 10
  37. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  38. 2025SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages?cited by 10

50 most cited of 205 All 205 (JSON)