# Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimation

arXiv: https://arxiv.org/abs/2608.03577 · HTML: https://mlsanity.com/paper/2608.03577/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Builds on (most cited first, 18 of 18)

- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [Uncertainty-Aware Machine Translation Evaluation](https://mlsanity.com/paper/2109.06352/) (2021)
- [Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors](https://mlsanity.com/paper/2310.16924/) (2023)
- [Measuring Uncertainty in Translation Quality Evaluation (TQE)](https://mlsanity.com/paper/2111.07699/) (2021)
- [IntelliCAT: Intelligent Machine Translation Post-Editing with Quality Estimation and Translation Suggestion](https://mlsanity.com/paper/2105.12172/) (2021)
- [Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation](https://mlsanity.com/paper/2410.10995/) (2024)
- [Better Quality Estimation for Low Resource Corpus Mining](https://mlsanity.com/paper/2203.08259/) (2022)
- [Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT quality](https://mlsanity.com/paper/1910.06204/) (2019)
- [Perturbation-based QE: An Explainable, Unsupervised Word-level Quality Estimation Method for Blackbox Machine Translation](https://mlsanity.com/paper/2305.07457/) (2023)
- [QE4PE: Word-level Quality Estimation for Human Post-Editing](https://mlsanity.com/paper/2503.03044/) (2025)
- [Tailoring Domain Adaptation for Machine Translation Quality Estimation](https://mlsanity.com/paper/2304.08891/) (2023)
- [Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimation](https://mlsanity.com/paper/2502.19941/) (2025)
- [Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement](https://mlsanity.com/paper/2505.23183/) (2025)
- [Intelligent Translation Memory Matching and Retrieval with Sentence Encoders](https://mlsanity.com/paper/2004.12894/) (2020)
- [Introducing Quality Estimation to Machine Translation Post-editing Workflow: An Empirical Study on Its Usefulness](https://mlsanity.com/paper/2507.16515/) (2025)
- [Penalizing Length: Uncovering Systematic Bias in Quality Estimation Metrics](https://mlsanity.com/paper/2510.22028/) (2025)
- [Automatically Generating Chinese Homophone Words to Probe Machine Translation Estimation Systems](https://mlsanity.com/paper/2503.16158/) (2025)
- [Semi-Synthetic Parallel Data for Translation Quality Estimation: A Case Study of Dataset Building for an Under-Resourced Language Pair](https://mlsanity.com/paper/2603.11743/) (2026)

## Built on by (most cited first, 1 of 1)

- [Mind the Gap: Exposing LLM Translation Blind Spots Using the AlphaMWE Multilingual Parallel Corpus](https://mlsanity.com/paper/2609.06634/) (2026)

## Abstract (arXiv)

Automation of Translation Quality Estimation (QE) has emerged as a widely discussed approach to managing translation quality at scale, and a growing number of tools and technologies have been released in pursuit of this goal. However, the proliferation of new QE systems has not always been accompanied by robust, transparent, and reproducible research and testing. This gap deserves critical scrutiny. This paper examines some fundamental limitations of the QE technology from both theoretical and empirical perspectives, arguing that current QE systems are structurally ill-equipped to serve as reliable standalone tools in real-world translation workflows. The reviewed evidence suggests that QE suffers from a range of interrelated and largely unresolved limitations. Most fundamentally, the evaluation of the quality of translation at the level of isolated segments is problematic because it tends to miss out on cohesion, coherence, and stylistic and rhetorical text features. In addition, empirical research documents several other limitations and flaws, including failure to generalize, systematic biases, overfitting and distribution collapse, performance gaps, error annotation challenges, and data scarcity. These are structural limitations arising from the complexity of human language and translation as a cognitive and communicative act - limitations that more data and better architectures have so far not overcome. Consequently, segment-level QE scores should not be used as a standalone basis for routing, release, or review bypass in production; we argue future work should focus on automating human evaluation grounded in MQM.