# Quality Estimation without Human-labeled Data

arXiv: https://arxiv.org/abs/2102.04020 · HTML: https://mlsanity.com/paper/2102.04020/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 8 Feb 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [OpenKiwi: An Open Source Framework for Quality Estimation](https://mlsanity.com/paper/1902.08646/) (2019)
- [Text Infilling](https://mlsanity.com/paper/1901.00158/) (2019)
- [eSCAPE: a Large-scale Synthetic Corpus for Automatic Post-Editing](https://mlsanity.com/paper/1803.07274/) (2018)
- [Unbabel's Participation in the WMT19 Translation Quality Estimation Shared Task](https://mlsanity.com/paper/1907.10352/) (2019)
- [Massively Multilingual Document Alignment with Cross-lingual Sentence-Mover's Distance](https://mlsanity.com/paper/2002.00761/) (2020)

## Built on by (most cited first, 15 of 15)

- [The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results](https://mlsanity.com/paper/2110.04392/) (2021)
- [Translation Error Detection as Rationale Extraction](https://mlsanity.com/paper/2108.12197/) (2021)
- [Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis](https://mlsanity.com/paper/2210.05035/) (2022)
- [As Little as Possible, as Much as Necessary: Detecting Over- and Undertranslations with Contrastive Conditioning](https://mlsanity.com/paper/2203.01927/) (2022)
- [Perturbation-based QE: An Explainable, Unsupervised Word-level Quality Estimation Method for Blackbox Machine Translation](https://mlsanity.com/paper/2305.07457/) (2023)
- [Beyond Glass-Box Features: Uncertainty Quantification Enhanced Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2109.07141/) (2021)
- [Are Generative Models Underconfident? Better Quality Estimation with Boosted Model Probability](https://mlsanity.com/paper/2502.11115/) (2025)
- [Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimation](https://mlsanity.com/paper/2502.19941/) (2025)
- [Rethink about the Word-level Quality Estimation for Machine Translation from Human Judgement](https://mlsanity.com/paper/2209.05695/) (2022)
- [Advancing Semi-Supervised Learning for Automatic Post-Editing: Data-Synthesis by Mask-Infilling with Erroneous Terms](https://mlsanity.com/paper/2204.03896/) (2022)
- [Mismatching-Aware Unsupervised Translation Quality Estimation For Low-Resource Languages](https://mlsanity.com/paper/2208.00463/) (2022)
- [Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications](https://mlsanity.com/paper/2109.08627/) (2021)
- [QUAK: A Synthetic Quality Estimation Dataset for Korean-English Neural Machine Translation](https://mlsanity.com/paper/2209.15285/) (2022)
- [Unsupervised Translation Quality Estimation Exploiting Synthetic Data and Pre-trained Multilingual Encoder](https://mlsanity.com/paper/2311.05117/) (2023)
- [Evaluating Language Translation Models by Playing Telephone](https://mlsanity.com/paper/2509.19611/) (2025)

## Abstract (arXiv)

Quality estimation aims to measure the quality of translated content without access to a reference translation. This is crucial for machine translation systems in real-world scenarios where high-quality translation is needed. While many approaches exist for quality estimation, they are based on supervised machine learning requiring costly human labelled data. As an alternative, we propose a technique that does not rely on examples from human-annotators and instead uses synthetic training data. We train off-the-shelf architectures for supervised quality estimation on our synthetic data and show that the resulting models achieve comparable performance to models trained on human-annotated data, both for sentence and word-level prediction.