# Mismatching-Aware Unsupervised Translation Quality Estimation For Low-Resource Languages

arXiv: https://arxiv.org/abs/2208.00463 · HTML: https://mlsanity.com/paper/2208.00463/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 53 references that are in the map.
- First posted on arXiv 31 Jul 2022.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)
- [Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks](https://mlsanity.com/paper/1909.00964/) (2019)
- [The IIT Bombay English-Hindi Parallel Corpus](https://mlsanity.com/paper/1710.02855/) (2017)
- [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://mlsanity.com/paper/2011.01536/) (2020)
- [OpenKiwi: An Open Source Framework for Quality Estimation](https://mlsanity.com/paper/1902.08646/) (2019)
- [MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset](https://mlsanity.com/paper/2010.04480/) (2020)
- [Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing](https://mlsanity.com/paper/1909.06775/) (2019)
- [Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study](https://mlsanity.com/paper/2009.14304/) (2020)
- [Context-Aware Cross-Lingual Mapping](https://mlsanity.com/paper/1903.03243/) (2019)
- [Quality Estimation without Human-labeled Data](https://mlsanity.com/paper/2102.04020/) (2021)
- [Unbabel's Participation in the WMT19 Translation Quality Estimation Shared Task](https://mlsanity.com/paper/1907.10352/) (2019)
- [Do Explicit Alignments Robustly Improve Multilingual Encoders?](https://mlsanity.com/paper/2010.02537/) (2020)
- [Zero-Shot Translation Quality Estimation with Explicit Cross-Lingual Patterns](https://mlsanity.com/paper/2010.04989/) (2020)
- [QEMind: Alibaba's Submission to the WMT21 Quality Estimation Shared Task](https://mlsanity.com/paper/2112.14890/) (2021)

## Built on by (most cited first, 1 of 1)

- [Towards Explainable Evaluation Metrics for Machine Translation](https://mlsanity.com/paper/2306.13041/) (2023)

## Abstract (arXiv)

Translation Quality Estimation (QE) is the task of predicting the quality of machine translation (MT) output without any reference. This task has gained increasing attention as an important component in the practical applications of MT. In this paper, we first propose XLMRScore, which is a cross-lingual counterpart of BERTScore computed via the XLM-RoBERTa (XLMR) model. This metric can be used as a simple unsupervised QE method, nevertheless facing two issues: firstly, the untranslated tokens leading to unexpectedly high translation scores, and secondly, the issue of mismatching errors between source and hypothesis tokens when applying the greedy matching in XLMRScore. To mitigate these issues, we suggest replacing untranslated words with the unknown token and the cross-lingual alignment of the pre-trained model to represent aligned words closer to each other, respectively. We evaluate the proposed method on four low-resource language pairs of the WMT21 QE shared task, as well as a new EnglishrightarrowPersian (En-Fa) test dataset introduced in this paper. Experiments show that our method could get comparable results with the supervised baseline for two zero-shot scenarios, i.e., with less than 0.01 difference in Pearson correlation, while outperforming unsupervised rivals in all the low-resource language pairs for above 8%, on average.