mlsanityData to 7 Oct 2026

Large Language Models Are State-of-the-Art Evaluators of Translation Quality

arXiv 2302.14520 ↗First posted 28 Feb 2023cs.CLcited by 416

Not on the map. Why?

Cited by 24 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We describe GEMBA, a GPT-based metric for assessment of translation quality, which works both with a reference translation and without. In our evaluation, we focus on zero-shot prompting, comparing four prompt variants in two modes, based on the availability of the reference. We investigate nine versions of GPT models, including ChatGPT and GPT-4. We show that our method for translation quality assessment only works with GPT~3.5 and larger models. Comparing to results from WMT22's Metrics shared task, our method achieves state-of-the-art accuracy in both modes when compared to MQM-based human labels. Our results are valid on the system level for all three WMT22 Metrics shared task language pairs, namely English into German, English into Russian, and Chinese into English. This provides a first glimpse into the usefulness of pre-trained, generative large language models for quality assessment of translations. We publicly release all our code and prompt templates used for the experiments described in this work, as well as all corresponding scoring results, to allow for external validation and reproducibility.

Key facts. Cited by 416 papers in the map (data through 7 Oct 2026). Builds on 9 of its 28 references that are in the map. First posted on arXiv 28 Feb 2023.

Builds on

9 of 28 references.

References · 9

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  6. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  7. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  8. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  9. 2022Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metriccited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 87; 2024: 129; 2025: 119; 2026: 81.

416 papers.

Sort:

By area:

  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  4. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  5. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  6. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  7. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  8. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  9. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  10. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  11. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  12. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
38 more
  1. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  2. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  3. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  4. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  5. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  6. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  7. 2023GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4cited by 143
  8. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  9. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 128
  10. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  11. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  12. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  13. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  14. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 94
  15. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  16. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 92
  17. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  18. 2023Document-Level Machine Translation with Large Language Modelscited by 90
  19. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  20. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 87
  21. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  22. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  23. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
  24. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  25. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  26. 2023Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibrationcited by 71
  27. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  28. 2023Table-GPT: Table-tuned GPT for Diverse Table Taskscited by 69
  29. 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
  30. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  31. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 66
  32. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 65
  33. 2023Mark My Words: Analyzing and Evaluating Language Model Watermarkscited by 62
  34. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60
  35. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 55
  36. 2023Benchmarking Machine Translation with Cultural Awarenesscited by 53
  37. 2023Calibrating LLM-Based Evaluatorcited by 53
  38. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50

50 most cited of 416 All 416 (JSON)