# Measuring Uncertainty in Translation Quality Evaluation (TQE)

arXiv: https://arxiv.org/abs/2111.07699 · HTML: https://mlsanity.com/paper/2111.07699/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 24 references that are in the map.
- First posted on arXiv 15 Nov 2021.

## Builds on (most cited first, 8 of 8)

- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Sampling Bias in Deep Active Classification: An Empirical Study](https://mlsanity.com/paper/1909.09389/) (2019)
- [A Systematic Characterization of Sampling Algorithms for Open-ended Language Generation](https://mlsanity.com/paper/2009.07243/) (2020)
- [Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods](https://mlsanity.com/paper/2105.03311/) (2021)
- [Neural Machine Translation Quality and Post-Editing Performance](https://mlsanity.com/paper/2109.05016/) (2021)
- [Chinese Character Decomposition for Neural MT with Multi-Word Expressions](https://mlsanity.com/paper/2104.04497/) (2021)
- [MultiMWE: Building a Multi-lingual Multi-Word Expression (MWE) Parallel Corpora](https://mlsanity.com/paper/2005.10583/) (2020)

## Built on by (most cited first, 16 of 16)

- [HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluation](https://mlsanity.com/paper/2112.13833/) (2021)
- [Finding Replicable Human Evaluations via Stable Ranking Probability](https://mlsanity.com/paper/2404.01474/) (2024)
- [Toward More Effective Human Evaluation for Machine Translation](https://mlsanity.com/paper/2204.05307/) (2022)
- [The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control](https://mlsanity.com/paper/2405.16969/) (2024)
- [An Overview on Machine Translation Evaluation](https://mlsanity.com/paper/2202.11027/) (2022)
- [Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning](https://mlsanity.com/paper/2210.06068/) (2022)
- [Topic Modelling of Swedish Newspaper Articles about Coronavirus: a Case Study using Latent Dirichlet Allocation Method](https://mlsanity.com/paper/2301.03029/) (2023)
- [Student's t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce](https://mlsanity.com/paper/2303.04526/) (2023)
- [A Comparative Study on Automatic Coding of Medical Letters with Explainability](https://mlsanity.com/paper/2407.13638/) (2024)
- [Non-Linear Scoring Model for Translation Quality Evaluation](https://mlsanity.com/paper/2511.13467/) (2025)
- [Ara-HOPE: Human-Centric Post-Editing Evaluation for Dialectal Arabic to Modern Standard Arabic Translation](https://mlsanity.com/paper/2512.21787/) (2025)
- [MetaHOPE: A Metaphor-Oriented Evaluation Framework for Analysing MT and LLM Translation Errors](https://mlsanity.com/paper/2607.00848/) (2026)
- [Quality Estimation of Machine Translated Texts based on Direct Evidence from Training Data](https://mlsanity.com/paper/2306.15399/) (2023)
- [MTUncertainty: Assessing the Need for Post-editing of Machine Translation Outputs by Fine-tuning OpenAI LLMs](https://mlsanity.com/paper/2308.00158/) (2023)
- [Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimation](https://mlsanity.com/paper/2608.03577/) (2026)
- [CANTONMT: Investigating Back-Translation and Model-Switch Mechanisms for Cantonese-English Neural Machine Translation](https://mlsanity.com/paper/2405.08172/) (2024)

## Abstract (arXiv)

From both human translators (HT) and machine translation (MT) researchers' point of view, translation quality evaluation (TQE) is an essential task. Translation service providers (TSPs) have to deliver large volumes of translations which meet customer specifications with harsh constraints of required quality level in tight time-frames and costs. MT researchers strive to make their models better, which also requires reliable quality evaluation. While automatic machine translation evaluation (MTE) metrics and quality estimation (QE) tools are widely available and easy to access, existing automated tools are not good enough, and human assessment from professional translators (HAP) are often chosen as the golden standard citehan-etal-2021-TQA. Human evaluations, however, are often accused of having low reliability and agreement. Is this caused by subjectivity or statistics is at play? How to avoid the entire text to be checked and be more efficient with TQE from cost and efficiency perspectives, and what is the optimal sample size of the translated text, so as to reliably estimate the translation quality of the entire material? This work carries out such motivated research to correctly estimate the confidence intervals citeBrownetal2001Interval depending on the sample size of the translated text, e.g. the amount of words or sentences, that needs to be processed on TQE workflow step for confident and reliable evaluation of overall translation quality. The methodology we applied for this work is from Bernoulli Statistical Distribution Modelling (BSDM) and Monte Carlo Sampling Analysis (MCSA).