mlsanityData to 7 Oct 2026

Unsupervised Quality Estimation for Neural Machine Translation

arXiv 2005.10608 ↗First posted 21 May 2020cs.CLcited by 210

Not on the map. Why?

Abstract (arXiv)

Quality Estimation (QE) is an important component in making Machine Translation (MT) useful in real-world applications, as it is aimed to inform the user on the quality of the MT output at test time. Existing approaches require large amounts of expert annotated data, computation and time for training. As an alternative, we devise an unsupervised approach to QE where no training or access to additional resources besides the MT system itself is required. Different from most of the current work that treats the MT system as a black box, we explore useful information that can be extracted from the MT system as a by-product of translation. By employing methods for uncertainty quantification, we achieve very good correlation with human judgments of quality, rivalling state-of-the-art supervised QE models. To evaluate our approach we collect the first dataset that enables work on both black-box and glass-box approaches to QE.

Key facts. Cited by 210 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 20; 2023: 27; 2024: 34; 2025: 56; 2026: 55.

210 papers.

Sort:

By area:

  1. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  2. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
  3. 2024Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantificationcited by 159
  4. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  5. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 130
  6. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  7. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  8. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 100
  9. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 79
  10. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 73
  11. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  12. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
38 more
  1. 2022Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translationcited by 60
  2. 2024Factual Confidence of LLMs: on Reliability and Robustness of Current Estimatorscited by 59
  3. 2025A Survey of Uncertainty Estimation Methods on Large Language Modelscited by 57
  4. 2023On Early Detection of Hallucinations in Factual Question Answeringcited by 54
  5. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  6. 2024MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMscited by 45
  7. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  8. 2025UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Makingcited by 36
  9. 2021Uncertainty-Aware Machine Translation Evaluationcited by 35
  10. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  11. 2025Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Homecited by 33
  12. 2025Real-Time Detection of Hallucinated Entities in Long-Form Generationcited by 33
  13. 2022Enhancing Cross-lingual Transfer by Manifold Mixupcited by 30
  14. 2023Do GPTs Produce Less Literal Translations?cited by 30
  15. 2025Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistencycited by 30
  16. 2023CLARA: Classifying and Disambiguating User Commands for Reliable Interactive Robotic Agentscited by 28
  17. 2024Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguitycited by 26
  18. 2024Do LLMs estimate uncertainty well in instruction-following?cited by 26
  19. 2025Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similaritycited by 26
  20. 2023On Uncertainty Calibration and Selective Generation in Probabilistic Neural Summarization: A Benchmark Studycited by 25
  21. 2024MAQA: Evaluating Uncertainty Quantification in LLMs Regarding Data Uncertaintycited by 25
  22. 2023Cousins Of The Vendi Score: A Family Of Similarity-Based Diversity Metrics For Science And Machine Learningcited by 24
  23. 2021Translation Quality Assessment: A Brief Survey on Manual and Automatic Methodscited by 23
  24. 2023RELIC: Investigating Large Language Model Responses using Self-Consistencycited by 23
  25. 2024Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMscited by 23
  26. 2021Translation Error Detection as Rationale Extractioncited by 22
  27. 2022Shifts 2.0: Extending The Dataset of Real Distributional Shiftscited by 22
  28. 2025Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Resultscited by 21
  29. 2022Learning to Scaffold: Optimizing Model Explanations for Teachingcited by 20
  30. 2021Uncertainty-Aware Balancing for Multilingual and Multi-Domain Neural Machine Translation Trainingcited by 19
  31. 2025Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMscited by 19
  32. 2024Word-Sequence Entropy: Towards Uncertainty Estimation in Free-Form Medical Question Answering Applications and Beyondcited by 18
  33. 2024Accounting for Sycophancy in Language Model Uncertainty Estimationcited by 18
  34. 2025Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Modelscited by 18
  35. 2023Conformal Autoregressive Generation: Beam Search with Coverage Guaranteescited by 16
  36. 2023An Empirical Study of Translation Hypothesis Ensembling with Large Language Modelscited by 16
  37. 2024Efficient Non-Parametric Uncertainty Quantification for Black-Box Large Language Models and Decision Planningcited by 16
  38. 2025Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?cited by 16

50 most cited of 210 All 210 (JSON)