mlsanityData to 7 Oct 2026

Convergences and Divergences between Automatic Assessment and Human Evaluation: Insights from Comparing ChatGPT-Generated Translation and Neural Machine Translation

arXiv 2401.05176 ↗First posted 10 Jan 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models have demonstrated parallel and even superior translation performance compared to neural machine translation (NMT) systems. However, existing comparative studies between them mainly rely on automated metrics, raising questions into the feasibility of these metrics and their alignment with human judgment. The present study investigates the convergences and divergences between automated metrics and human evaluation in assessing the quality of machine translation from ChatGPT and three NMT systems. To perform automatic assessment, four automated metrics are employed, while human evaluation incorporates the DQF-MQM error typology and six rubrics. Notably, automatic assessment and human evaluation converge in measuring formal fidelity (e.g., error rates), but diverge when evaluating semantic and pragmatic fidelity, with automated metrics failing to capture the improvement of ChatGPT's translation brought by prompt engineering. These results underscore the indispensable role of human judgment in evaluating the performance of advanced translation tools at the current stage.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 46 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

14 of 46 references.

References · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  5. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  6. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  7. 2022What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cited by 149
  8. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  9. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
  10. 2023Document-Level Machine Translation with Large Language Modelscited by 90
  11. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  12. 2023Large language models effectively leverage document-level context for literary translation, but critical errors persistcited by 80
2 more
  1. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  2. 2023Augmenting Large Language Model Translators via Translation Memoriescited by 21

Built on by

Not cited yet.