# MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluators

arXiv: https://arxiv.org/abs/2409.14335 · HTML: https://mlsanity.com/paper/2409.14335/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 22 Sep 2024.

## Builds on (most cited first, 30 of 33)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection](https://mlsanity.com/paper/2310.10482/) (2023)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4](https://mlsanity.com/paper/2310.13988/) (2023)
- [Towards Making the Most of ChatGPT for Machine Translation](https://mlsanity.com/paper/2303.13780/) (2023)
- [Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration](https://mlsanity.com/paper/2305.14324/) (2023)
- [A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods](https://mlsanity.com/paper/2104.00054/) (2021)
- [Leveraging GPT-4 for Automatic Translation Post-Editing](https://mlsanity.com/paper/2305.14878/) (2023)
- [Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies](https://mlsanity.com/paper/2401.06760/) (2024)
- [LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback](https://mlsanity.com/paper/2311.09336/) (2023)
- [Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model](https://mlsanity.com/paper/2401.12873/) (2024)
- [IndicMT Eval: A Dataset to Meta-Evaluate Machine Translation metrics for Indian Languages](https://mlsanity.com/paper/2212.10180/) (2022)
- [The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics](https://mlsanity.com/paper/2310.19792/) (2023)
- [Guiding Large Language Models to Post-Edit Machine Translation with Error Annotations](https://mlsanity.com/paper/2404.07851/) (2024)
- [Aligning Neural Machine Translation Models: Human Feedback in Training and Inference](https://mlsanity.com/paper/2311.09132/) (2023)
- [Toward Human-Like Evaluation for Natural Language Generation with Error Analysis](https://mlsanity.com/paper/2212.10179/) (2022)
- [xTower: A Multilingual LLM for Explaining and Correcting Translation Errors](https://mlsanity.com/paper/2406.19482/) (2024)

## Built on by (most cited first, 22 of 22)

- [How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMs](https://mlsanity.com/paper/2410.18697/) (2024)
- [Specification-Aware Machine Translation and Evaluation for Purpose Alignment](https://mlsanity.com/paper/2509.17559/) (2025)
- [BatchGEMBA: Token-Efficient Machine Translation Evaluation with Batched Prompting and Prompt Compression](https://mlsanity.com/paper/2503.02756/) (2025)
- [HiMATE: A Hierarchical Multi-Agent Framework for Machine Translation Evaluation](https://mlsanity.com/paper/2505.16281/) (2025)
- [LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation](https://mlsanity.com/paper/2604.18490/) (2026)
- [Testing LLMs' Capabilities in Annotating Translations Based on an Error Typology Designed for LSP Translation: First Experiments with ChatGPT](https://mlsanity.com/paper/2504.15052/) (2025)
- [Beyond Literal Translation: Evaluating Cultural Effectiveness in Social Media UGC](https://mlsanity.com/paper/2605.25626/) (2026)
- [MockConf: A Student Interpretation Dataset: Analysis, Word- and Span-level Alignment and Baselines](https://mlsanity.com/paper/2506.04848/) (2025)
- [SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languages](https://mlsanity.com/paper/2509.20557/) (2025)
- [Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation](https://mlsanity.com/paper/2512.07540/) (2025)
- [Remedy-R: Generative Reasoning for Machine Translation Evaluation without Error Annotations](https://mlsanity.com/paper/2512.18906/) (2025)
- [Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation](https://mlsanity.com/paper/2605.24904/) (2026)
- [Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages](https://mlsanity.com/paper/2607.02235/) (2026)
- [Improving Term Evaluation in Machine Translation: Variation Matters](https://mlsanity.com/paper/2609.08779/) (2026)
- [TULUN: Transparent and Adaptable Low-resource Machine Translation](https://mlsanity.com/paper/2505.18683/) (2025)
- [To Write or to Automate Linguistic Prompts, That Is the Question](https://mlsanity.com/paper/2603.25169/) (2026)
- [CompactQE: Interpretable Translation Quality Estimation via Small Open-Weight LLMs](https://mlsanity.com/paper/2605.15763/) (2026)
- [Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports](https://mlsanity.com/paper/2606.18797/) (2026)
- [Machine Translation and Post-Editing: Comparative Evaluation of Different MT Systems and Post-Editor Groups in Specialised Translation](https://mlsanity.com/paper/2606.23002/) (2026)
- [TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation](https://mlsanity.com/paper/2608.02975/) (2026)
- [Evaluating Communicative Success in Machine-Translated Conversation](https://mlsanity.com/paper/2609.19885/) (2026)
- [FACET at WMT 2026 Automated Translation Quality Evaluation Task](https://mlsanity.com/paper/2610.00096/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown significant potential as judges for Machine Translation (MT) quality assessment, providing both scores and fine-grained feedback. Although approaches such as GEMBA-MQM have shown state-of-the-art performance on reference-free evaluation, the predicted errors do not align well with those annotated by human, limiting their interpretability as feedback signals. To enhance the quality of error annotations predicted by LLM evaluators, we introduce a universal and training-free framework, MQM-APE, based on the idea of filtering out non-impactful errors by Automatically Post-Editing (APE) the original translation based on each error, leaving only those errors that contribute to quality improvement. Specifically, we prompt the LLM to act as 1) textitevaluator to provide error annotations, 2) textitpost-editor to determine whether errors impact quality improvement and 3) textitpairwise quality verifier as the error filter. Experiments show that our approach consistently improves both the reliability and quality of error spans against GEMBA-MQM, across eight LLMs in both high- and low-resource languages. Orthogonal to trained approaches, MQM-APE complements translation-specific evaluators such as Tower, highlighting its broad applicability. Further analysis confirms the effectiveness of each module and offers valuable insights into evaluator design and LLMs selection.