mlsanityData to 7 Oct 2026

QE4PE: Word-level Quality Estimation for Human Post-Editing

arXiv 2503.03044 ↗First posted 4 Mar 2025cs.CL cs.HCcited by 6

Not on the map. Why?

Abstract (arXiv)

Word-level quality estimation (QE) methods aim to detect erroneous spans in machine translations, which can direct and facilitate human post-editing. While the accuracy of word-level QE systems has been assessed extensively, their usability and downstream influence on the speed, quality and editing choices of human post-editing remain understudied. In this study, we investigate the impact of word-level QE on machine translation (MT) post-editing in a realistic setting involving 42 professional post-editors across two translation directions. We compare four error-span highlight modalities, including supervised and uncertainty-based word-level QE methods, for identifying potential errors in the outputs of a state-of-the-art neural MT model. Post-editing effort and productivity are estimated from behavioral logs, while quality improvements are assessed by word- and segment-level human annotation. We find that domain, language and editors' speed are critical factors in determining highlights' effectiveness, with modest differences between human-made and automated QE highlights underlining a gap between accuracy and usability in professional workflows.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 35 of its 72 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

35 of 72 references.

References · 35

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  2. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  3. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  4. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  5. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  6. 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 176
  7. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
  8. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  9. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  10. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  11. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  12. 2022Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Bettercited by 75
23 more
  1. 2024Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translationcited by 62
  2. 2019OpenKiwi: An Open Source Framework for Quality Estimationcited by 49
  3. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  4. 2021The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Resultscited by 27
  5. 2022ACES: Translation Accuracy Challenge Sets for Evaluating Machine Translation Metricscited by 25
  6. 2024How Culture Shapes What People Want From AIcited by 23
  7. 2023Physician Detection of Clinical Harm in Machine Translation: Quality Estimation Aids in Reliance and Backtranslation Identifies Critical Errorscited by 22
  8. 2024Fine-Tuned Machine Translation Metrics Struggle in Unseen Domainscited by 21
  9. 2023Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completionscited by 20
  10. 2024AI-Assisted Human Evaluation of Machine Translationcited by 20
  11. 2023Domain Terminology Integration into Machine Translation: Leveraging Large Language Modelscited by 17
  12. 2024Can Automatic Metrics Assess High-Quality Translations?cited by 16
  13. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  14. 2022Investigating the Benefits of Free-Form Rationalescited by 13
  15. 2021Neural Machine Translation Quality and Post-Editing Performancecited by 12
  16. 2022DivEMT: Neural Machine Translation Post-Editing Effort Across Typologically Diverse Languagescited by 11
  17. 2021Backtranslation Feedback Improves User Confidence in MT, Not Qualitycited by 10
  18. 2024Enhanced Hallucination Detection in Neural Machine Translation through Simple Detector Aggregationcited by 9
  19. 2025Translation in the Hands of Many:Centering Lay Users in Machine Translation Interactionscited by 8
  20. 2024Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Datacited by 4
  21. 2023ACES: Translation Accuracy Challenge Sets at WMT 2023cited by 3
  22. 2020The Impact of Text Presentation on Translator Performancecited by 1
  23. 2023Predicting Human Translation Difficulty with Neural Machine Translationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreementcited by 4
  2. 2026Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translationcited by 1
  3. 2025Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluationcited by 0
  4. 2026Smarter edits? Post-editing with error highlights and translation suggestionscited by 0
  5. 2026Measuring User's Mental Models of Speech Translation in Human-AI Collaborationcited by 0
  6. 2026Looking under the Wrong Lamppost: On the Limitations of Automated Translation Quality Estimationcited by 0