# An Analysis on Automated Metrics for Evaluating Japanese-English Chat Translation

arXiv: https://arxiv.org/abs/2412.18190 · HTML: https://mlsanity.com/paper/2412.18190/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- First posted on arXiv 24 Dec 2024.

## Abstract (arXiv)

This paper analyses how traditional baseline metrics, such as BLEU and TER, and neural-based methods, such as BERTScore and COMET, score several NMT models performance on chat translation and how these metrics perform when compared to human-annotated scores. The results show that for ranking NMT models in chat translations, all metrics seem consistent in deciding which model outperforms the others. This implies that traditional baseline metrics, which are faster and simpler to use, can still be helpful. On the other hand, when it comes to better correlation with human judgment, neural-based metrics outperform traditional metrics, with COMET achieving the highest correlation with the human-annotated score on a chat translation. However, we show that even the best metric struggles when scoring English translations from sentences with anaphoric zero-pronoun in Japanese.