# An Overview on Machine Translation Evaluation

arXiv: https://arxiv.org/abs/2202.11027 · HTML: https://mlsanity.com/paper/2202.11027/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 141 references that are in the map.
- First posted on arXiv 22 Feb 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation](https://mlsanity.com/paper/1808.07048/) (2018)
- [Translationese in Machine Translation Evaluation](https://mlsanity.com/paper/1906.09833/) (2019)
- [A Set of Recommendations for Assessing Human-Machine Parity in Language Translation](https://mlsanity.com/paper/2004.01694/) (2020)
- [Translation Quality Assessment: A Brief Survey on Manual and Automatic Methods](https://mlsanity.com/paper/2105.03311/) (2021)
- [Measuring Uncertainty in Translation Quality Evaluation (TQE)](https://mlsanity.com/paper/2111.07699/) (2021)
- [Pairwise Neural Machine Translation Evaluation](https://mlsanity.com/paper/1912.03135/) (2019)
- [Chinese Character Decomposition for Neural MT with Multi-Word Expressions](https://mlsanity.com/paper/2104.04497/) (2021)
- [cushLEPOR: customising hLEPOR metric using Optuna for higher agreement with human judgments or pre-trained language model LaBSE](https://mlsanity.com/paper/2108.09484/) (2021)
- [LEPOR: An Augmented Machine Translation Evaluation Metric](https://mlsanity.com/paper/1703.08748/) (2017)
- [Neural Machine Translation Advised by Statistical Machine Translation](https://mlsanity.com/paper/1610.05150/) (2016)

## Built on by (most cited first, 4 of 4)

- [Examining Large Pre-Trained Language Models for Machine Translation: What You Don't Know About It](https://mlsanity.com/paper/2209.07417/) (2022)
- [Topic Modelling of Swedish Newspaper Articles about Coronavirus: a Case Study using Latent Dirichlet Allocation Method](https://mlsanity.com/paper/2301.03029/) (2023)
- [Student's t-Distribution: On Measuring the Inter-Rater Reliability When the Observations are Scarce](https://mlsanity.com/paper/2303.04526/) (2023)
- [Asynchronous and Segmented Bidirectional Encoding for NMT](https://mlsanity.com/paper/2402.14849/) (2024)

## Abstract (arXiv)

Since the 1950s, machine translation (MT) has become one of the important tasks of AI and development, and has experienced several different periods and stages of development, including rule-based methods, statistical methods, and recently proposed neural network-based learning methods. Accompanying these staged leaps is the evaluation research and development of MT, especially the important role of evaluation methods in statistical translation and neural translation research. The evaluation task of MT is not only to evaluate the quality of machine translation, but also to give timely feedback to machine translation researchers on the problems existing in machine translation itself, how to improve and how to optimise. In some practical application fields, such as in the absence of reference translations, the quality estimation of machine translation plays an important role as an indicator to reveal the credibility of automatically translated target languages. This report mainly includes the following contents: a brief history of machine translation evaluation (MTE), the classification of research methods on MTE, and the the cutting-edge progress, including human evaluation, automatic evaluation, and evaluation of evaluation methods (meta-evaluation). Manual evaluation and automatic evaluation include reference-translation based and reference-translation independent participation; automatic evaluation methods include traditional n-gram string matching, models applying syntax and semantics, and deep learning models; evaluation of evaluation methods includes estimating the credibility of human evaluations, the reliability of the automatic evaluation, the reliability of the test set, etc. Advances in cutting-edge evaluation methods include task-based evaluation, using pre-trained language models based on big data, and lightweight optimisation models using distillation techniques.