# HOPE: A Task-Oriented and Human-Centric Evaluation Framework Using Professional Post-Editing Towards More Effective MT Evaluation arXiv: https://arxiv.org/abs/2112.13833 ยท HTML: https://mlsanity.com/paper/2112.13833/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Builds on 6 of its 21 references that are in the map. - First posted on arXiv 27 Dec 2021. ## Builds on (most cited first, 6 of 6) - [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021) - [Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods](https://mlsanity.com/paper/2105.03311/) (2021) - [Neural versus Phrase-Based Machine Translation Quality: a Case Study](https://mlsanity.com/paper/1608.04631/) (2016) - [Measuring Uncertainty in Translation Quality Evaluation (TQE)](https://mlsanity.com/paper/2111.07699/) (2021) - [cushLEPOR: customising hLEPOR metric using Optuna for higher agreement with human judgments or pre-trained language model LaBSE](https://mlsanity.com/paper/2108.09484/) (2021) - [MultiMWE: Building a Multi-lingual Multi-Word Expression (MWE) Parallel Corpora](https://mlsanity.com/paper/2005.10583/) (2020) ## Built on by (most cited first, 18 of 18) - [Toward Human-Like Evaluation for Natural Language Generation with Error Analysis](https://mlsanity.com/paper/2212.10179/) (2022) - [The Multi-Range Theory of Translation Quality Measurement: MQM scoring models and Statistical Quality Control](https://mlsanity.com/paper/2405.16969/) (2024) - [Investigating Large Language Models and Control Mechanisms to Improve Text Readability of Biomedical Abstracts](https://mlsanity.com/paper/2309.13202/) (2023) - [Neural Machine Translation of Clinical Text: An Empirical Investigation into Multilingual Pre-Trained Language Models and Transfer-Learning](https://mlsanity.com/paper/2312.07250/) (2023) - [Investigating Massive Multilingual Pre-Trained Machine Translation Models for Clinical Domain via Transfer Learning](https://mlsanity.com/paper/2210.06068/) (2022) - [Topic Modelling of Swedish Newspaper Articles about Coronavirus: a Case Study using Latent Dirichlet Allocation Method](https://mlsanity.com/paper/2301.03029/) (2023) - [Student's t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce](https://mlsanity.com/paper/2303.04526/) (2023) - [Non-Linear Scoring Model for Translation Quality Evaluation](https://mlsanity.com/paper/2511.13467/) (2025) - [Ara-HOPE: Human-Centric Post-Editing Evaluation for Dialectal Arabic to Modern Standard Arabic Translation](https://mlsanity.com/paper/2512.21787/) (2025) - [MetaHOPE: A Metaphor-Oriented Evaluation Framework for Analysing MT and LLM Translation Errors](https://mlsanity.com/paper/2607.00848/) (2026) - [MTUncertainty: Assessing the Need for Post-editing of Machine Translation Outputs by Fine-tuning OpenAI LLMs](https://mlsanity.com/paper/2308.00158/) (2023) - [Large Language Models for Biomedical Text Simplification: Promising But Not There Yet](https://mlsanity.com/paper/2408.03871/) (2024) - [MaLei at the PLABA Track of TREC 2024: RoBERTa for Term Replacement -- LLaMA3.1 and GPT-4o for Complete Abstract Adaptation](https://mlsanity.com/paper/2411.07381/) (2024) - [HilMeMe: A Human-in-the-Loop Machine Translation Evaluation Metric Looking into Multi-Word Expressions](https://mlsanity.com/paper/2211.05201/) (2022) - [CANTONMT: Investigating Back-Translation and Model-Switch Mechanisms for Cantonese-English Neural Machine Translation](https://mlsanity.com/paper/2405.08172/) (2024) - [Data-Efficient Domain Adaptation for LLM-based MT using Contrastive Preference Optimization](https://mlsanity.com/paper/2510.27556/) (2025) - [Towards End-to-End Multilingual Metaphor Processing: Integrating Detection, Translation, and Evaluation](https://mlsanity.com/paper/2608.04260/) (2026) - [SIREN: Situation-aware Index for Response and Emergency ActioNability in Crisis Translation](https://mlsanity.com/paper/2610.16646/) (2026) ## Abstract (arXiv) Traditional automatic evaluation metrics for machine translation have been widely criticized by linguists due to their low accuracy, lack of transparency, focus on language mechanics rather than semantics, and low agreement with human quality evaluation. Human evaluations in the form of MQM-like scorecards have always been carried out in real industry setting by both clients and translation service providers (TSPs). However, traditional human translation quality evaluations are costly to perform and go into great linguistic detail, raise issues as to inter-rater reliability (IRR) and are not designed to measure quality of worse than premium quality translations. In this work, we introduce HOPE, a task-oriented and human-centric evaluation framework for machine translation output based on professional post-editing annotations. It contains only a limited number of commonly occurring error types, and use a scoring model with geometric progression of error penalty points (EPPs) reflecting error severity level to each translation unit. The initial experimental work carried out on English-Russian language pair MT outputs on marketing content type of text from highly technical domain reveals that our evaluation framework is quite effective in reflecting the MT output quality regarding both overall system-level performance and segment-level transparency, and it increases the IRR for error type interpretation. The approach has several key advantages, such as ability to measure and compare less than perfect MT output from different systems, ability to indicate human perception of quality, immediate estimation of the labor effort required to bring MT output to premium quality, low-cost and faster application, as well as higher IRR. Our experimental data is available at urlhttps://github.com/lHan87/HOPE.