# An Exploratory Analysis of Multilingual Word-Level Quality Estimation with Cross-Lingual Transformers

arXiv: https://arxiv.org/abs/2106.00143 · HTML: https://mlsanity.com/paper/2106.00143/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [Multilingual Offensive Language Identification with Cross-lingual Embeddings](https://mlsanity.com/paper/2010.05324/) (2020)
- [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://mlsanity.com/paper/2011.01536/) (2020)
- [Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation](https://mlsanity.com/paper/1708.09803/) (2017)
- [OpenKiwi: An Open Source Framework for Quality Estimation](https://mlsanity.com/paper/1902.08646/) (2019)
- [MUDES: Multilingual Detection of Offensive Spans](https://mlsanity.com/paper/2102.09665/) (2021)
- [TransQuest at WMT2020: Sentence-Level Direct Assessment](https://mlsanity.com/paper/2010.05318/) (2020)

## Built on by (most cited first, 18 of 18)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translation](https://mlsanity.com/paper/2202.10062/) (2022)
- [Towards Explainable Evaluation Metrics for Natural Language Generation](https://mlsanity.com/paper/2203.11131/) (2022)
- [Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks](https://mlsanity.com/paper/2305.10284/) (2023)
- [The Glass Ceiling of Automatic Evaluation in Natural Language Generation](https://mlsanity.com/paper/2208.14585/) (2022)
- [What do Large Language Models Need for Machine Translation Evaluation?](https://mlsanity.com/paper/2410.03278/) (2024)
- [BERT(s) to Detect Multiword Expressions](https://mlsanity.com/paper/2208.07832/) (2022)
- [DTW at Qur'an QA 2022: Utilising Transfer Learning with Transformers for Question Answering in a Low-resource Domain](https://mlsanity.com/paper/2205.06025/) (2022)
- [Towards Fine-Grained Information: Identifying the Type and Location of Translation Errors](https://mlsanity.com/paper/2302.08975/) (2023)
- [REFeREE: A REference-FREE Model-Based Metric for Text Simplification](https://mlsanity.com/paper/2403.17640/) (2024)
- [Giving the Old a Fresh Spin: Quality Estimation-Assisted Constrained Decoding for Automatic Post-Editing](https://mlsanity.com/paper/2501.17265/) (2025)
- [Rethink about the Word-level Quality Estimation for Machine Translation from Human Judgement](https://mlsanity.com/paper/2209.05695/) (2022)
- [SurreyAI 2023 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2312.00525/) (2023)
- [A Multi-task Learning Framework for Evaluating Machine Translation of Emotion-loaded User-generated Content](https://mlsanity.com/paper/2410.03277/) (2024)
- [What are the best systems? New perspectives on NLP Benchmarking](https://mlsanity.com/paper/2202.03799/) (2022)
- [Transformer-based Detection of Multiword Expressions in Flower and Plant Names](https://mlsanity.com/paper/2209.08016/) (2022)
- [QUAK: A Synthetic Quality Estimation Dataset for Korean-English Neural Machine Translation](https://mlsanity.com/paper/2209.15285/) (2022)
- [BMX: Boosting Natural Language Generation Metrics with Explainability](https://mlsanity.com/paper/2212.10469/) (2022)

## Abstract (arXiv)

Most studies on word-level Quality Estimation (QE) of machine translation focus on language-specific models. The obvious disadvantages of these approaches are the need for labelled data for each language pair and the high cost required to maintain several language-specific models. To overcome these problems, we explore different approaches to multilingual, word-level QE. We show that these QE models perform on par with the current language-specific models. In the cases of zero-shot and few-shot QE, we demonstrate that it is possible to accurately predict word-level quality for any given new language pair from models trained on other language pairs. Our findings suggest that the word-level QE models based on powerful pre-trained transformers that we propose in this paper generalise well across languages, making them more useful in real-world scenarios.