mlsanityData to 14 Oct 2026

GEMBA-MQM: Detecting Translation Quality Error Spans with GPT-4

arXiv 2310.13988 ↗First posted 21 Oct 2023cs.CLcited by 146

Not on the map. Why?

Abstract (arXiv)

This paper introduces GEMBA-MQM, a GPT-based evaluation metric designed to detect translation quality errors, specifically for the quality estimation setting without the need for human reference translations. Based on the power of large language models (LLM), GEMBA-MQM employs a fixed three-shot prompting technique, querying the GPT-4 model to mark error quality spans. Compared to previous works, our method has language-agnostic prompts, thus avoiding the need for manual prompt preparation for new languages. While preliminary results indicate that GEMBA-MQM achieves state-of-the-art accuracy for system ranking, we advise caution when using it in academic works to demonstrate improvements over other methods due to its dependence on the proprietary, black-box GPT model.

Key facts. Cited by 146 papers in the map (data through 14 Oct 2026). Builds on 9 of its 27 references that are in the map. First posted on arXiv 21 Oct 2023.

Builds on

9 of 27 references.

References · 9

  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  2. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  3. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 338
  4. 2023How is ChatGPT's behavior changing over time?cited by 270
  5. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 163
  6. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 158
  7. 2021Learning Compact Metrics for MTcited by 93
  8. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  9. 2022Alibaba-Translate China's Submission for WMT 2022 Metrics Shared Taskcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 41; 2025: 39; 2026: 66.

146 papers.

Sort:

By area:

  1. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  2. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  3. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  4. 2024Adapting Large Language Models for Document-Level Machine Translationcited by 59
  5. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  6. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 48
  7. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  8. 2024TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinementcited by 39
  9. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  10. 2025How to Select Datapoints for Efficient Human Evaluation of NLG Models?cited by 23
  11. 2024MERA: A Comprehensive LLM Evaluation in Russiancited by 22
  12. 2024Guiding Large Language Models to Post-Edit Machine Translation with Error Annotationscited by 22
38 more
  1. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  2. 2024Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In!cited by 21
  3. 2024AI-Assisted Human Evaluation of Machine Translationcited by 20
  4. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  5. 2024Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translationcited by 16
  6. 2024Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generationcited by 16
  7. 2024Can Automatic Metrics Assess High-Quality Translations?cited by 16
  8. 2024PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluationcited by 16
  9. 2024MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferencescited by 15
  10. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
  11. 2024M-MAD: Multidimensional Multi-Agent Debate for Advanced Machine Translation Evaluationcited by 13
  12. 2025SwiLTra-Bench: The Swiss Legal Translation Benchmarkcited by 11
  13. 2024Creative and Context-Aware Translation of East Asian Idioms with GPT-4cited by 10
  14. 2024Beyond Correlation: Interpretable Evaluation of Machine Translation Metricscited by 10
  15. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  16. 2025Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answeringcited by 10
  17. 2025LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answeringcited by 10
  18. 2025Has Machine Translation Evaluation Achieved Human Parity? The Human Reference and the Limits of Progresscited by 10
  19. 2025MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Taskcited by 10
  20. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translationcited by 8
  21. 2024xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metricscited by 7
  22. 2024Towards Automatic Evaluation for Image Transcreationcited by 7
  23. 2025LLMs as Span Annotators: A Comparative Study of LLMs and Humanscited by 7
  24. 2025AskQE: Question Answering as Automatic Evaluation for Machine Translationcited by 7
  25. 2024How Good is Zero-Shot MT Evaluation for Low Resource Indian Languages?cited by 6
  26. 2024What do Large Language Models Need for Machine Translation Evaluation?cited by 6
  27. 2024MetaMetrics-MT: Tuning Meta-Metrics for Machine Translation via Human Preference Calibrationcited by 6
  28. 2025QE4PE: Word-level Quality Estimation for Human Post-Editingcited by 6
  29. 2026Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluationcited by 6
  30. 2025EnDive: A Cross-Dialect Benchmark for Fairness and Performance in Large Language Modelscited by 5
  31. 2026UbuntuGuard: A Culturally-Grounded Policy Benchmark for Equitable AI Safety in African Languagescited by 5
  32. 2024Reranking Laws for Language Generation: A Communication-Theoretic Perspectivecited by 4
  33. 2024PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metricscited by 4
  34. 2025Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimationcited by 4
  35. 2025OpeNLGauge: An Explainable Metric for NLG Evaluation with Open-Weights LLMscited by 4
  36. 2025Specification-Aware Machine Translation and Evaluation for Purpose Alignmentcited by 4
  37. 2025DITING: A Multi-Agent Evaluation Framework for Benchmarking Web Novel Translationcited by 4
  38. 2026Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigationcited by 4

50 most cited of 146 All 146 (JSON)