# Evaluating Machine Translation Performance on Chinese Idioms with a Blacklist Method

arXiv: https://arxiv.org/abs/1711.07646 · HTML: https://mlsanity.com/paper/1711.07646/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2017.

## Built on by (most cited first, 12 of 12)

- [Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation](https://mlsanity.com/paper/2205.15301/) (2022)
- [Translate Meanings, Not Just Words: IdiomKB's Role in Optimizing Idiomatic Translation with Language Models](https://mlsanity.com/paper/2308.13961/) (2023)
- [Improving LLM Abilities in Idiomatic Translation](https://mlsanity.com/paper/2407.03518/) (2024)
- [Do We Need Language-Specific Fact-Checking Models? The Case of Chinese](https://mlsanity.com/paper/2401.15498/) (2024)
- [Crossing the Threshold: Idiomatic Machine Translation through Retrieval Augmentation and Loss Weighting](https://mlsanity.com/paper/2310.07081/) (2023)
- [PETCI: A Parallel English Translation Dataset of Chinese Idioms](https://mlsanity.com/paper/2202.09509/) (2022)
- [Automatic Evaluation and Analysis of Idioms in Neural Machine Translation](https://mlsanity.com/paper/2210.04545/) (2022)
- [Chengyu-Bench: Benchmarking Large Language Models for Chinese Idiom Understanding and Use](https://mlsanity.com/paper/2506.18105/) (2025)
- [Hierarchical Phrase-based Sequence-to-Sequence Learning](https://mlsanity.com/paper/2211.07906/) (2022)
- [Chinese Idiom Paraphrasing](https://mlsanity.com/paper/2204.07555/) (2022)
- [Evaluating LLMs on Chinese Idiom Translation](https://mlsanity.com/paper/2508.10421/) (2025)
- [The Comparison of Translationese in Machine Translation and Human Transation in terms of Translation Relations](https://mlsanity.com/paper/2404.08661/) (2024)

## Abstract (arXiv)

Idiom translation is a challenging problem in machine translation because the meaning of idioms is non-compositional, and a literal (word-by-word) translation is likely to be wrong. In this paper, we focus on evaluating the quality of idiom translation of MT systems. We introduce a new evaluation method based on an idiom-specific blacklist of literal translations, based on the insight that the occurrence of any blacklisted words in the translation output indicates a likely translation error. We introduce a dataset, CIBB (Chinese Idioms Blacklists Bank), and perform an evaluation of a state-of-the-art Chinese-English neural MT system. Our evaluation confirms that a sizable number of idioms in our test set are mistranslated (46.1%), that literal translation error is a common error type, and that our blacklist method is effective at identifying literal translation errors.