mlsanityData to 7 Oct 2026

The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation

arXiv 2308.07286 ↗First posted 14 Aug 2023cs.CL cs.LGcited by 95

Not on the map. Why?

Abstract (arXiv)

Automatic evaluation of machine translation (MT) is a critical tool driving the rapid iterative development of MT systems. While considerable progress has been made on estimating a single scalar quality score, current metrics lack the informativeness of more detailed schemes that annotate individual errors, such as Multidimensional Quality Metrics (MQM). In this paper, we help fill this gap by proposing AutoMQM, a prompting technique which leverages the reasoning and in-context learning capabilities of large language models (LLMs) and asks them to identify and categorize errors in translations. We start by evaluating recent LLMs, such as PaLM and PaLM-2, through simple score prediction prompting, and we study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores (with particularly large gains for larger models) while providing interpretability through error spans that align with human annotations.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 20 of its 36 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

20 of 36 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  11. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  12. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
8 more
  1. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  2. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  3. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  4. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  5. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  6. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  7. 2023Multi-Dimensional Evaluation of Text Summarization with In-Context Learningcited by 44
  8. 2022T5Score: Discriminative Fine-tuning of Generative Evaluation Metricscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 34; 2025: 32; 2026: 19.

95 papers.

Sort:

By area:

  1. 2024Self-Rewarding Language Modelscited by 654
  2. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  3. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  4. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  5. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  6. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 92
  7. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  8. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  9. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  10. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  11. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  12. 2023LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedbackcited by 39
38 more
  1. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  2. 2024How Susceptible are LLMs to Influence in Prompts?cited by 32
  3. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 29
  4. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 27
  5. 2024Self-rationalization improves LLM as a fine-grained judgecited by 25
  6. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  7. 2023Assessing Large Language Models on Climate Informationcited by 22
  8. 2024Guiding Large Language Models to Post-Edit Machine Translation with Error Annotationscited by 22
  9. 2025Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contaminationcited by 22
  10. 2024Fine-Tuned Machine Translation Metrics Struggle in Unseen Domainscited by 21
  11. 2024Timo: Towards Better Temporal Reasoning for Language Modelscited by 21
  12. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  13. 2024xTower: A Multilingual LLM for Explaining and Correcting Translation Errorscited by 17
  14. 2024Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translationcited by 16
  15. 2024PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluationcited by 15
  16. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  17. 2024MT-Ranker: Reference-free machine translation evaluation by inter-system rankingcited by 14
  18. 2023Which is better? Exploring Prompting Strategy For LLM-based Metricscited by 13
  19. 2024CriticEval: Evaluating Large Language Model as Criticcited by 13
  20. 2024Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Modelscited by 13
  21. 2024M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluationcited by 13
  22. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
  23. 2024Creative and Context-Aware Translation of East Asian Idioms with GPT-4cited by 10
  24. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  25. 2025Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progresscited by 10
  26. 2025MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Taskcited by 10
  27. 2023Training and Meta-Evaluating Machine Translation Evaluation Metrics at the Paragraph Levelcited by 9
  28. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 9
  29. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translationcited by 8
  30. 2023Leveraging Word Guessing Games to Assess the Intelligence of Large Language Modelscited by 7
  31. 2025When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languagescited by 7
  32. 2025Enhancing Human Evaluation in Machine Translation with Comparative Judgmentcited by 7
  33. 2025LLMs as Span Annotators: A Comparative Study of LLMs and Humanscited by 7
  34. 2025AskQE: Question Answering as Automatic Evaluation for Machine Translationcited by 7
  35. 2024What do Large Language Models Need for Machine Translation Evaluation?cited by 6
  36. 2025QE4PE: Word-level Quality Estimation for Human Post-Editingcited by 6
  37. 2025Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Modelscited by 6
  38. 2025Extending Automatic Machine Translation Evaluation to Book-Length Documentscited by 6

50 most cited of 95 All 95 (JSON)