mlsanityData to 14 Oct 2026

Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task

arXiv 2309.11925 ↗First posted 21 Sep 2023cs.CLcited by 88

Not on the map. Why?

Abstract (arXiv)

We present the joint contribution of Unbabel and Instituto Superior T'ecnico to the WMT 2023 Shared Task on Quality Estimation (QE). Our team participated on all tasks: sentence- and word-level quality prediction (task 1) and fine-grained error span detection (task 2). For all tasks, we build on the COMETKIWI-22 model (Rei et al., 2022b). Our multilingual approaches are ranked first for all tasks, reaching state-of-the-art performance for quality estimation at word-, span- and sentence-level granularity. Compared to the previous state-of-the-art COMETKIWI-22, we show large improvements in correlation with human judgements (up to 10 Spearman points). Moreover, we surpass the second-best multilingual submission to the shared-task with up to 3.8 absolute points.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Builds on 9 of its 17 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

9 of 17 references.

References · 9

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  2. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  3. 2018Deep contextualized word representationscited by 2,290
  4. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  5. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  6. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 207
  7. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  8. 2020MLQE-PE: A Multilingual Quality Estimation and Post-Editing Datasetcited by 47
  9. 2022UniTE: Unified Translation Evaluationcited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 30; 2025: 29; 2026: 25.

88 papers.

Sort:

By area:

  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  2. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 265
  3. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  4. 2025Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Studycited by 38
  5. 2024mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generationcited by 32
  6. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  7. 2024How Much Data is Enough Data? Fine-Tuning Large Language Models for In-House Translation: Performance Evaluation Across Multiple Dataset Sizescited by 17
  8. 2024Can Automatic Metrics Assess High-Quality Translations?cited by 16
  9. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  10. 2024Mitigating Metric Bias in Minimum Bayes Risk Decodingcited by 16
  11. 2024MT-Ranker: Reference-free machine translation evaluation by inter-system rankingcited by 14
  12. 2025Compositional Translation: A Novel LLM-based Approach for Low-resource Machine Translationcited by 14
38 more
  1. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
  2. 2023xDial-Eval: A Multilingual Open-Domain Dialogue Evaluation Benchmarkcited by 13
  3. 2024Improving Minimum Bayes Risk Decoding with Multi-Promptcited by 13
  4. 2025SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translationcited by 13
  5. 2024Dialectal Toxicity Detection: Evaluating LLM-as-a-Judge Consistency Across Language Varietiescited by 12
  6. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
  7. 2025Multilingual Contextualization of Large Language Models for Document-Level Machine Translationcited by 11
  8. 2024Pitfalls and Outlooks in Using COMETcited by 10
  9. 2024Beyond Correlation: Interpretable Evaluation of Machine Translation Metricscited by 10
  10. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  11. 2024SpeechQE: Estimating the Quality of Direct Speech Translationcited by 10
  12. 2025Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progresscited by 10
  13. 2025MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Taskcited by 10
  14. 2023Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisationscited by 9
  15. 2025Remedy: Learning Machine Translation Evaluation from Human Preferences with Reward Modelingcited by 9
  16. 2025Multilingual Performance Biases of Large Language Models in Educationcited by 9
  17. 2024Modeling User Preferences with Automatic Metrics: Creating a High-Quality Preference Dataset for Machine Translationcited by 8
  18. 2024LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistenciescited by 7
  19. 2025When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languagescited by 7
  20. 2025Semantic Agreement Enables Efficient Open-Ended LLM Cascadescited by 7
  21. 2025Evaluation of Multilingual Image Captioning: How far can we get with CLIP models?cited by 5
  22. 2025TACTIC: Translation Agents with Cognitive-Theoretic Interactive Collaborationcited by 5
  23. 2025ConsistencyChecker: Tree-based Evaluation of LLM Generalization Capabilitiescited by 5
  24. 2024Instruction-tuned Large Language Models for Machine Translation in the Medical Domaincited by 4
  25. 2024Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysiscited by 4
  26. 2024MojoBench: Language Modeling and Benchmarks for Mojocited by 4
  27. 2025Translate Smart, not Hard: Cascaded Translation Systems with Quality-Aware Deferralcited by 4
  28. 2025Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimationcited by 4
  29. 2025Translate, then Detect: Leveraging Machine Translation for Cross-Lingual Toxicity Classificationcited by 4
  30. 2025Feeding Two Birds or Favoring One? Adequacy-Fluency Tradeoffs in Evaluation and Meta-Evaluation of Machine Translationcited by 4
  31. 2025M2PO: Multi-Perspective Multi-Pair Preference Optimization for Machine Translationcited by 4
  32. 2025MultiScript30k: Leveraging Multilingual Embeddings to Extend Cross Script Parallel Datacited by 4
  33. 2026Scaling Model and Data for Multilingual Machine Translation with Open Large Language Modelscited by 4
  34. 2024Towards Red Teaming in Multimodal and Multilingual Translationcited by 3
  35. 2024IKUN for WMT24 General MT Task: LLMs Are here for Multilingual Machine Translationcited by 3
  36. 2024Post-edits Are Preferences Toocited by 3
  37. 2025CRPO: Confidence-Reward Driven Preference Optimization for Machine Translationcited by 3
  38. 2026NeoAMT: Neologism-Aware Agentic Machine Translation with Reinforcement Learningcited by 3

50 most cited of 88 All 88 (JSON)