# Unsupervised Quality Estimation for Neural Machine Translation

arXiv: https://arxiv.org/abs/2005.10608 · HTML: https://mlsanity.com/paper/2005.10608/

## Key facts

- Cited by 211 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2020.

## Built on by (most cited first, 30 of 211)

- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://mlsanity.com/paper/2210.02410/) (2022)
- [Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification](https://mlsanity.com/paper/2403.04696/) (2024)
- [Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models](https://mlsanity.com/paper/2307.01379/) (2023)
- [LM-Polygraph: Uncertainty Estimation for Language Models](https://mlsanity.com/paper/2311.07383/) (2023)
- [Benchmarking LLMs via Uncertainty Quantification](https://mlsanity.com/paper/2401.12794/) (2024)
- [Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph](https://mlsanity.com/paper/2406.15627/) (2024)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)
- [LUQ: Long-text Uncertainty Quantification for LLMs](https://mlsanity.com/paper/2403.20279/) (2024)
- [Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach](https://mlsanity.com/paper/2404.15993/) (2024)
- [A Survey of Multimodal Large Language Model from A Data-centric Perspective](https://mlsanity.com/paper/2405.16640/) (2024)
- [A Survey of Uncertainty Estimation Methods on Large Language Models](https://mlsanity.com/paper/2503.00172/) (2025)
- [Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2208.05309/) (2022)
- [Uncertainty in Natural Language Generation: From Theory to Applications](https://mlsanity.com/paper/2307.15703/) (2023)
- [Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators](https://mlsanity.com/paper/2406.13415/) (2024)
- [On Early Detection of Hallucinations in Factual Question Answering](https://mlsanity.com/paper/2312.14183/) (2023)
- [What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability](https://mlsanity.com/paper/2305.11707/) (2023)
- [MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs](https://mlsanity.com/paper/2402.11756/) (2024)
- [Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection](https://mlsanity.com/paper/2301.07779/) (2023)
- [UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making](https://mlsanity.com/paper/2506.17419/) (2025)
- [Uncertainty-Aware Machine Translation Evaluation](https://mlsanity.com/paper/2109.06352/) (2021)
- [A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs](https://mlsanity.com/paper/2412.03324/) (2024)
- [Real-Time Detection of Hallucinated Entities in Long-Form Generation](https://mlsanity.com/paper/2509.03531/) (2025)
- [Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home](https://mlsanity.com/paper/2501.12835/) (2025)
- [Enhancing Cross-lingual Transfer by Manifold Mixup](https://mlsanity.com/paper/2205.04182/) (2022)
- [Do GPTs Produce Less Literal Translations?](https://mlsanity.com/paper/2305.16806/) (2023)
- [CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agents](https://mlsanity.com/paper/2306.10376/) (2023)
- [Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency](https://mlsanity.com/paper/2502.04964/) (2025)
- [Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity](https://mlsanity.com/paper/2506.00245/) (2025)
- [Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguity](https://mlsanity.com/paper/2402.06529/) (2024)

## Abstract (arXiv)

Quality Estimation (QE) is an important component in making Machine Translation (MT) useful in real-world applications, as it is aimed to inform the user on the quality of the MT output at test time. Existing approaches require large amounts of expert annotated data, computation and time for training. As an alternative, we devise an unsupervised approach to QE where no training or access to additional resources besides the MT system itself is required. Different from most of the current work that treats the MT system as a black box, we explore useful information that can be extracted from the MT system as a by-product of translation. By employing methods for uncertainty quantification, we achieve very good correlation with human judgments of quality, rivalling state-of-the-art supervised QE models. To evaluate our approach we collect the first dataset that enables work on both black-box and glass-box approaches to QE.