# QE4PE: Word-level Quality Estimation for Human Post-Editing

arXiv: https://arxiv.org/abs/2503.03044 · HTML: https://mlsanity.com/paper/2503.03044/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 72 references that are in the map.
- First posted on arXiv 4 Mar 2025.

## Builds on (most cited first, 30 of 35)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing](https://mlsanity.com/paper/2004.14564/) (2020)
- [The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation](https://mlsanity.com/paper/2308.07286/) (2023)
- [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://mlsanity.com/paper/2105.00572/) (2021)
- [Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better](https://mlsanity.com/paper/2212.08597/) (2022)
- [Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.11580/) (2024)
- [OpenKiwi: An Open Source Framework for Quality Estimation](https://mlsanity.com/paper/1902.08646/) (2019)
- [Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection](https://mlsanity.com/paper/2301.07779/) (2023)
- [The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Results](https://mlsanity.com/paper/2110.04392/) (2021)
- [ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metrics](https://mlsanity.com/paper/2210.15615/) (2022)
- [How Culture Shapes What People Want From AI](https://mlsanity.com/paper/2403.05104/) (2024)
- [Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errors](https://mlsanity.com/paper/2310.16924/) (2023)
- [Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains](https://mlsanity.com/paper/2402.18747/) (2024)
- [Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions](https://mlsanity.com/paper/2302.07248/) (2023)
- [AI-Assisted Human Evaluation of Machine Translation](https://mlsanity.com/paper/2406.12419/) (2024)
- [Domain Terminology Integration into Machine Translation: Leveraging Large Language Models](https://mlsanity.com/paper/2310.14451/) (2023)
- [Can Automatic Metrics Assess High-Quality Translations?](https://mlsanity.com/paper/2405.18348/) (2024)
- [Towards Explainable Evaluation Metrics for Machine Translation](https://mlsanity.com/paper/2306.13041/) (2023)
- [Investigating the Benefits of Free-Form Rationales](https://mlsanity.com/paper/2206.11083/) (2022)
- [Neural Machine Translation Quality and Post-Editing Performance](https://mlsanity.com/paper/2109.05016/) (2021)
- [DivEMT: Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languages](https://mlsanity.com/paper/2205.12215/) (2022)
- [Backtranslation Feedback Improves User Confidence in MT, Not Quality](https://mlsanity.com/paper/2104.05688/) (2021)
- [Enhanced Hallucination Detection in Neural Machine Translation through Simple Detector Aggregation](https://mlsanity.com/paper/2402.13331/) (2024)

## Built on by (most cited first, 6 of 6)

- [Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreement](https://mlsanity.com/paper/2505.23183/) (2025)
- [Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation](https://mlsanity.com/paper/2512.07540/) (2025)
- [Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translation](https://mlsanity.com/paper/2606.17234/) (2026)
- [Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimation](https://mlsanity.com/paper/2608.03577/) (2026)
- [Smarter edits? Post-editing with error highlights and translation suggestions](https://mlsanity.com/paper/2605.21135/) (2026)
- [Measuring User's Mental Models of Speech Translation in Human-AI Collaboration](https://mlsanity.com/paper/2606.24644/) (2026)

## Abstract (arXiv)

Word-level quality estimation (QE) methods aim to detect erroneous spans in machine translations, which can direct and facilitate human post-editing. While the accuracy of word-level QE systems has been assessed extensively, their usability and downstream influence on the speed, quality and editing choices of human post-editing remain understudied. In this study, we investigate the impact of word-level QE on machine translation (MT) post-editing in a realistic setting involving 42 professional post-editors across two translation directions. We compare four error-span highlight modalities, including supervised and uncertainty-based word-level QE methods, for identifying potential errors in the outputs of a state-of-the-art neural MT model. Post-editing effort and productivity are estimated from behavioral logs, while quality improvements are assessed by word- and segment-level human annotation. We find that domain, language and editors' speed are critical factors in determining highlights' effectiveness, with modest differences between human-made and automated QE highlights underlining a gap between accuracy and usability in professional workflows.