# Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task

arXiv: https://arxiv.org/abs/2309.11925 · HTML: https://mlsanity.com/paper/2309.11925/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 17 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 9 of 9)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://mlsanity.com/paper/1602.02068/) (2016)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://mlsanity.com/paper/2308.07286/) (2023)
- [MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset](https://mlsanity.com/paper/2010.04480/) (2020)
- [UniTE: Unified Translation Evaluation](https://mlsanity.com/paper/2204.13346/) (2022)

## Built on by (most cited first, 30 of 88)

- [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://mlsanity.com/paper/2401.08417/) (2024)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study](https://mlsanity.com/paper/2502.02481/) (2025)
- [mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generation](https://mlsanity.com/paper/2410.15037/) (2024)
- [The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics](https://mlsanity.com/paper/2310.19792/) (2023)
- [How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizes](https://mlsanity.com/paper/2409.03454/) (2024)
- [Can Automatic Metrics Assess High-Quality Translations?](https://mlsanity.com/paper/2405.18348/) (2024)
- [QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation](https://mlsanity.com/paper/2406.00049/) (2024)
- [Mitigating Metric Bias in Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2411.03524/) (2024)
- [MT-Ranker: Reference-free machine translation evaluation by inter-system ranking](https://mlsanity.com/paper/2401.17099/) (2024)
- [Compositional Translation: A Novel LLM-based Approach for Low-resource Machine Translation](https://mlsanity.com/paper/2503.04554/) (2025)
- [Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models](https://mlsanity.com/paper/2504.01001/) (2025)
- [xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmark](https://mlsanity.com/paper/2310.08958/) (2023)
- [Improving Minimum Bayes Risk Decoding with Multi-Prompt](https://mlsanity.com/paper/2407.15343/) (2024)
- [SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation](https://mlsanity.com/paper/2505.16637/) (2025)
- [Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varieties](https://mlsanity.com/paper/2411.10954/) (2024)
- [Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings](https://mlsanity.com/paper/2411.05986/) (2024)
- [Multilingual Contextualization of Large Language Models for Document-Level Machine Translation](https://mlsanity.com/paper/2504.12140/) (2025)
- [Pitfalls and Outlooks in Using COMET](https://mlsanity.com/paper/2408.15366/) (2024)
- [Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics](https://mlsanity.com/paper/2410.05183/) (2024)
- [Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation](https://mlsanity.com/paper/2410.10995/) (2024)
- [SpeechQE: Estimating the Quality of Direct Speech Translation](https://mlsanity.com/paper/2410.21485/) (2024)
- [Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progress](https://mlsanity.com/paper/2506.19571/) (2025)
- [MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Task](https://mlsanity.com/paper/2510.24707/) (2025)
- [Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations](https://mlsanity.com/paper/2312.13871/) (2023)
- [Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modeling](https://mlsanity.com/paper/2504.13630/) (2025)
- [Multilingual Performance Biases of Large Language Models in Education](https://mlsanity.com/paper/2504.17720/) (2025)
- [Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translation](https://mlsanity.com/paper/2410.07779/) (2024)
- [LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies](https://mlsanity.com/paper/2412.15035/) (2024)

## Abstract (arXiv)

We present the joint contribution of Unbabel and Instituto Superior T'ecnico to the WMT 2023 Shared Task on Quality Estimation (QE). Our team participated on all tasks: sentence- and word-level quality prediction (task 1) and fine-grained error span detection (task 2). For all tasks, we build on the COMETKIWI-22 model (Rei et al., 2022b). Our multilingual approaches are ranked first for all tasks, reaching state-of-the-art performance for quality estimation at word-, span- and sentence-level granularity. Compared to the previous state-of-the-art COMETKIWI-22, we show large improvements in correlation with human judgements (up to 10 Spearman points). Moreover, we surpass the second-best multilingual submission to the shared-task with up to 3.8 absolute points.